2024-06-15T09:24:34.247 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-15T09:24:34.247 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi/7756600 --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} --name yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi/7756600/orig.config.yaml 2024-06-15T09:24:34.269 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T09:24:37.967 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi/7756600 branch: squid-release description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7756600' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid-release ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd workunit: branch: squid sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8772 sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvKpqIyheQ3e5Ctj2wiOXsB1iEEKfPN3qnww5GtKuNMzuYvxOxmQY5HwjtF1BHydbJKbg59pZDv8DLuLYmjwLM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-14_18:48:07 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-15T09:24:37.967 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa; will attempt to use it 2024-06-15T09:24:37.968 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks 2024-06-15T09:24:37.968 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-15T09:24:37.970 INFO:teuthology.task.internal:Checking packages... 2024-06-15T09:24:37.991 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd' 2024-06-15T09:24:37.991 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T09:24:37.991 INFO:teuthology.packaging:ref: None 2024-06-15T09:24:37.992 INFO:teuthology.packaging:tag: None 2024-06-15T09:24:37.992 INFO:teuthology.packaging:branch: squid-release 2024-06-15T09:24:37.992 INFO:teuthology.packaging:sha1: c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-15T09:24:37.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid-release 2024-06-15T09:24:38.134 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2541-gc65be810-1jammy 2024-06-15T09:24:38.136 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-15T09:24:38.143 INFO:teuthology.task.internal:no buildpackages task found 2024-06-15T09:24:38.143 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-15T09:24:38.150 INFO:teuthology.task.internal:Saving configuration 2024-06-15T09:24:38.158 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-15T09:24:38.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-15T09:24:38.192 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi/7756600', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-15 09:19:03.084279', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO+dIZ0SrDmGMxDC0m+LERVeKcUo2GhTaObri+P0s6HN'} 2024-06-15T09:24:38.192 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-15T09:24:38.198 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-06-15T09:24:38.198 INFO:teuthology.run_tasks:Running task console_log... 2024-06-15T09:24:38.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f18980dfac0>, signals=[15]) 2024-06-15T09:24:38.229 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-15T09:24:38.239 INFO:teuthology.task.internal:Opening connections... 2024-06-15T09:24:38.239 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-06-15T09:24:38.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:24:38.321 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-15T09:24:38.327 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-06-15T09:24:38.423 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-06-15T09:24:38.423 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-06-15T09:24:38.469 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-06-15T09:24:38.469 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2024-06-15T09:24:38.469 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="22.04" 2024-06-15T09:24:38.469 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=jammy 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-15T09:24:38.470 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=jammy 2024-06-15T09:24:38.471 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-06-15T09:24:38.498 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-15T09:24:38.507 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-15T09:24:38.512 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-15T09:24:38.512 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-06-15T09:24:38.516 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-15T09:24:38.522 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-15T09:24:38.522 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-06-15T09:24:38.568 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-15T09:24:38.636 INFO:teuthology.run_tasks:Running task kernel... 2024-06-15T09:24:38.652 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-15T09:24:38.652 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-15T09:24:38.653 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-15T09:24:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T09:24:38.657 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-06-15T09:24:38.705 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-56-generic 2024-06-15T09:24:38.706 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.15.0-56-generic 2024-06-15T09:24:38.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-06-15T09:24:38.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-06-15T09:24:39.046 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T09:24:39.058 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-15T09:24:39.142 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-06-15T09:24:39.148 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-06-15T09:24:39.281 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [642 kB] 2024-06-15T09:24:39.358 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,731 kB] 2024-06-15T09:24:39.401 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [318 kB] 2024-06-15T09:24:39.407 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-06-15T09:24:39.407 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,990 kB] 2024-06-15T09:24:39.434 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-06-15T09:24:39.434 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [338 kB] 2024-06-15T09:24:39.438 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T09:24:39.439 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [707 kB] 2024-06-15T09:24:39.445 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,086 kB] 2024-06-15T09:24:39.454 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [251 kB] 2024-06-15T09:24:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-06-15T09:24:39.482 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-06-15T09:24:39.483 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.0 kB] 2024-06-15T09:24:39.483 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.7 kB] 2024-06-15T09:24:39.484 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-06-15T09:24:39.484 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-06-15T09:24:39.484 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-06-15T09:24:39.485 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-06-15T09:24:39.490 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-06-15T09:24:39.511 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-06-15T09:24:39.511 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-06-15T09:24:39.511 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.3 kB] 2024-06-15T09:24:39.512 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-06-15T09:24:39.590 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [480 kB] 2024-06-15T09:24:39.862 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,517 kB] 2024-06-15T09:24:40.031 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [259 kB] 2024-06-15T09:24:40.043 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-06-15T09:24:40.043 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,933 kB] 2024-06-15T09:24:40.125 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-06-15T09:24:40.126 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [329 kB] 2024-06-15T09:24:40.134 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-06-15T09:24:40.135 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [607 kB] 2024-06-15T09:24:40.149 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [858 kB] 2024-06-15T09:24:40.166 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [166 kB] 2024-06-15T09:24:40.181 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-06-15T09:24:40.195 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-06-15T09:24:40.196 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-06-15T09:24:40.277 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-06-15T09:24:40.277 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-06-15T09:24:45.150 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 14.0 MB in 2s (7,329 kB/s) 2024-06-15T09:24:46.462 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:24:46.479 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T09:24:46.541 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:24:46.748 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-06-15T09:24:46.749 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-06-15T09:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T09:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T09:24:46.914 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T09:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T09:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-06-15T09:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout: python2.7-minimal 2024-06-15T09:24:46.915 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T09:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-06-15T09:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T09:24:46.916 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-generic linux-image-5.15.0-112-generic 2024-06-15T09:24:46.917 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-5.15.0-112-generic linux-modules-extra-5.15.0-112-generic 2024-06-15T09:24:46.918 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-06-15T09:24:46.918 INFO:teuthology.orchestra.run.smithi116.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-06-15T09:24:46.918 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-06-15T09:24:46.918 INFO:teuthology.orchestra.run.smithi116.stdout: thermald 2024-06-15T09:24:46.975 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-06-15T09:24:46.975 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-5.15.0-112 linux-headers-5.15.0-112-generic 2024-06-15T09:24:46.975 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.15.0-112-generic linux-modules-5.15.0-112-generic 2024-06-15T09:24:46.975 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-extra-5.15.0-112-generic 2024-06-15T09:24:46.976 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2024-06-15T09:24:46.977 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-generic linux-image-generic 2024-06-15T09:24:47.156 INFO:teuthology.orchestra.run.smithi116.stdout:3 upgraded, 5 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T09:24:47.156 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 113 MB of archives. 2024-06-15T09:24:47.156 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 583 MB of additional disk space will be used. 2024-06-15T09:24:47.156 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-112-generic amd64 5.15.0-112.122 [22.6 MB] 2024-06-15T09:24:48.567 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-112-generic amd64 5.15.0-112.122 [11.5 MB] 2024-06-15T09:24:48.915 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-112-generic amd64 5.15.0-112.122 [63.9 MB] 2024-06-15T09:24:51.057 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.112.112 [1,692 B] 2024-06-15T09:24:51.057 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.112.112 [2,510 B] 2024-06-15T09:24:51.058 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112 all 5.15.0-112.122 [12.3 MB] 2024-06-15T09:24:51.438 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-112-generic amd64 5.15.0-112.122 [2,904 kB] 2024-06-15T09:24:51.522 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.112.112 [2,360 B] 2024-06-15T09:24:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 113 MB in 5s (25.0 MB/s) 2024-06-15T09:24:52.099 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-5.15.0-112-generic. 2024-06-15T09:24:54.127 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-06-15T09:24:54.131 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../0-linux-modules-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T09:24:54.253 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:24:56.913 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-image-5.15.0-112-generic. 2024-06-15T09:24:56.934 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../1-linux-image-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T09:24:57.027 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:24:57.516 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-112-generic. 2024-06-15T09:24:57.536 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T09:24:57.599 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:04.188 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../3-linux-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T09:25:04.298 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T09:25:04.657 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T09:25:04.793 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T09:25:05.054 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-112. 2024-06-15T09:25:05.066 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../5-linux-headers-5.15.0-112_5.15.0-112.122_all.deb ... 2024-06-15T09:25:05.104 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T09:25:11.041 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-112-generic. 2024-06-15T09:25:11.077 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../6-linux-headers-5.15.0-112-generic_5.15.0-112.122_amd64.deb ... 2024-06-15T09:25:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:15.134 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.112.112_amd64.deb ... 2024-06-15T09:25:15.393 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-generic (5.15.0.112.112) over (5.15.0.56.54) ... 2024-06-15T09:25:15.784 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-112 (5.15.0-112.122) ... 2024-06-15T09:25:15.955 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:16.082 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-generic (5.15.0.112.112) ... 2024-06-15T09:25:16.209 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:17.579 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-06-15T09:25:17.579 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-06-15T09:25:17.579 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-112-generic 2024-06-15T09:25:17.580 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-112-generic 2024-06-15T09:25:17.737 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-extra-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:19.154 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:21.073 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-generic (5.15.0.112.112) ... 2024-06-15T09:25:21.199 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-generic (5.15.0.112.112) ... 2024-06-15T09:25:21.342 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for linux-image-5.15.0-112-generic (5.15.0-112.122) ... 2024-06-15T09:25:21.412 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-06-15T09:25:21.412 INFO:teuthology.orchestra.run.smithi116.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-112-generic 2024-06-15T09:25:35.746 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-06-15T09:25:35.746 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub' 2024-06-15T09:25:35.779 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T09:25:35.810 INFO:teuthology.orchestra.run.smithi116.stdout:Generating grub configuration file ... 2024-06-15T09:25:36.117 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T09:25:36.131 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T09:25:36.433 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T09:25:36.439 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T09:25:36.505 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T09:25:36.510 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T09:25:36.697 INFO:teuthology.orchestra.run.smithi116.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T09:25:36.698 INFO:teuthology.orchestra.run.smithi116.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T09:25:36.698 INFO:teuthology.orchestra.run.smithi116.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T09:25:36.752 INFO:teuthology.orchestra.run.smithi116.stdout:done 2024-06-15T09:25:37.192 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.112.112 2024-06-15T09:25:37.212 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T09:25:37.213 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T09:25:37.213 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-06-15T09:25:37.213 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-06-15T09:25:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-06-15T09:25:37.213 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-06-15T09:25:37.213 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-112-generic 2024-06-15T09:25:37.213 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-15T09:25:37.213 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-15T09:25:37.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-15T09:25:37.213 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-06-15T09:25:37.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-06-15T09:25:37.395 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-06-15T09:25:37.681 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T09:25:37.681 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T09:25:37.744 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-06-15T09:25:38.106 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T09:25:38.268 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 128 kB in 1s (185 kB/s) 2024-06-15T09:25:39.244 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:25:39.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-06-15T09:25:39.326 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:25:39.520 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-06-15T09:25:39.520 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-06-15T09:25:39.684 INFO:teuthology.orchestra.run.smithi116.stdout:linux-image-generic is already the newest version (5.15.0.112.112). 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T09:25:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T09:25:39.686 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T09:25:39.725 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 0 newly installed, 0 to remove and 326 not upgraded. 2024-06-15T09:25:39.727 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-06-15T09:25:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-06-15T09:25:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-06-15T09:25:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.112.112 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-112), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-112-generic, linux-modules-extra-5.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-06-15T09:25:39.743 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-06-15T09:25:39.744 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-06-15T09:25:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.u8sAZO1fXo 2024-06-15T09:25:39.788 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub/grub.cfg /tmp/tmp.u8sAZO1fXo 2024-06-15T09:25:39.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.u8sAZO1fXo 2024-06-15T09:25:39.957 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.u8sAZO1fXo is 10KB 2024-06-15T09:25:39.971 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.u8sAZO1fXo 2024-06-15T09:25:39.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-06-15T09:25:40.032 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:25:40.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-06-15T09:25:40.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-06-15T09:25:40.123 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-112-generic 2024-06-15T09:25:40.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-grub 2024-06-15T09:25:41.398 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2024-06-15T09:25:41.399 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-06-15T09:25:41.403 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-06-15T09:25:41.586 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-112-generic 2024-06-15T09:25:41.592 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-112-generic 2024-06-15T09:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-06-15T09:25:41.893 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-06-15T09:25:41.934 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-06-15T09:25:41.940 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-06-15T09:25:42.058 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-06-15T09:25:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-06-15T09:25:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-06-15T09:25:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-06-15T09:25:42.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-06-15T09:26:12.110 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-15T09:26:12.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:26:12.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:26:30.431 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-06-15T09:26:39.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:26:39.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:26:42.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-06-15T09:26:54.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:26:54.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:27:12.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-06-15T09:27:27.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:27:27.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:27:28.470 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-06-15T09:27:29.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:27:29.286 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-112-generic"... 2024-06-15T09:27:29.287 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-06-15T09:27:29.333 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-112-generic 2024-06-15T09:27:29.334 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-112-generic vs 5.15.0-112-generic 2024-06-15T09:27:29.334 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-15T09:27:29.334 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-06-15T09:27:30.334 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-15T09:27:30.335 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-15T09:27:30.412 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-06-15T09:27:30.443 DEBUG:teuthology.parallel:result is None 2024-06-15T09:27:30.443 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-15T09:27:30.452 INFO:teuthology.task.internal:Creating test directory... 2024-06-15T09:27:30.452 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-15T09:27:30.458 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-15T09:27:30.473 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-15T09:27:30.486 INFO:teuthology.task.internal:Creating archive directory... 2024-06-15T09:27:30.487 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-15T09:27:30.524 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-15T09:27:30.535 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-15T09:27:30.535 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-15T09:27:30.579 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:27:30.586 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:27:30.587 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-15T09:27:30.598 INFO:teuthology.task.internal:Configuring sudo... 2024-06-15T09:27:30.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-15T09:27:30.639 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-15T09:27:30.652 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-15T09:27:30.652 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-15T09:27:30.684 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T09:27:30.731 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T09:27:30.778 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:27:30.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-15T09:27:30.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-06-15T09:27:30.937 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-15T09:27:30.946 INFO:teuthology.task.internal:Starting timer... 2024-06-15T09:27:30.946 INFO:teuthology.run_tasks:Running task pcp... 2024-06-15T09:27:30.963 INFO:teuthology.run_tasks:Running task selinux... 2024-06-15T09:27:30.977 DEBUG:teuthology.task.selinux:Excluding smithi116: OS 'ubuntu' does not support SELinux 2024-06-15T09:27:30.977 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-15T09:27:30.977 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-06-15T09:27:30.977 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-15T09:27:30.977 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-15T09:27:30.989 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-15T09:27:30.992 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-15T09:27:31.102 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-15T09:27:31.112 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-15T09:27:31.113 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-15T09:34:29.146 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2024-06-15T09:34:29.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:34:29.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T09:34:29.232 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-06-15T09:34:29.310 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-06-15T09:34:29.310 INFO:teuthology.run_tasks:Running task clock... 2024-06-15T09:34:29.323 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-15T09:34:29.323 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-15T09:34:29.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T09:34:29.381 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Command line: ntpd -gq 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: ---------------------------------------------------- 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: ntp-4 is maintained by Network Time Foundation, 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: corporation. Support and training for ntp-4 are 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: available at https://www.nwtime.org/support 2024-06-15T09:34:29.382 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: ---------------------------------------------------- 2024-06-15T09:34:29.383 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: proto: precision = 0.060 usec (-24) 2024-06-15T09:34:29.383 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: basedate set to 2022-02-04 2024-06-15T09:34:29.383 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: gps base set to 2022-02-06 (week 2196) 2024-06-15T09:34:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T09:34:29.384 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-06-15T09:34:29.384 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: restrict ::: KOD does nothing without LIMITED. 2024-06-15T09:34:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:restrict ::: KOD does nothing without LIMITED. 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen and drop on 0 v6wildcard [::]:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen normally on 2 lo 127.0.0.1:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen normally on 3 enp3s0f1 172.21.15.116:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen normally on 4 lo [::1]:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:3f67%5]:123 2024-06-15T09:34:29.385 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:29 ntpd[17562]: Listening on routing socket on fd #22 for interface updates 2024-06-15T09:34:30.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:30.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:30 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:31.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:31.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:31.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:31 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:32.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:32.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:32.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:32 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:33 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:34.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:34.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:34.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:34.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:34.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:34.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:34 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:35.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:35 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:36.382 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-06-15T09:34:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: can't open /var/log/ntpstats/rawstats.20240615: Permission denied 2024-06-15T09:34:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-06-15T09:34:36.383 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: can't open /var/log/ntpstats/peerstats.20240615: Permission denied 2024-06-15T09:34:36.383 INFO:teuthology.orchestra.run.smithi116.stdout:15 Jun 09:34:36 ntpd[17562]: ntpd: time slew -0.001679 s 2024-06-15T09:34:36.383 INFO:teuthology.orchestra.run.smithi116.stdout:ntpd: time slew -0.001679s 2024-06-15T09:34:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-06-15T09:34:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:15 Jun 09:34:36 ntpd[17562]: can't open /var/log/ntpstats/loopstats.20240615: Permission denied 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T09:34:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-06-15T09:34:36.436 INFO:teuthology.run_tasks:Running task install... 2024-06-15T09:34:36.473 DEBUG:teuthology.task.install:project ceph 2024-06-15T09:34:36.473 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd'}} 2024-06-15T09:34:36.474 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd'} 2024-06-15T09:34:36.474 INFO:teuthology.task.install:Using flavor: default 2024-06-15T09:34:36.479 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-15T09:34:36.479 INFO:teuthology.task.install:extra packages: [] 2024-06-15T09:34:36.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-key list | grep Ceph 2024-06-15T09:34:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-06-15T09:34:36.564 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-06-15T09:34:36.564 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph.com (release key) 2024-06-15T09:34:36.565 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-06-15T09:34:36.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-15T09:34:36.714 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default/ 2024-06-15T09:34:36.714 INFO:teuthology.task.install.deb:Package version is 19.0.0-2541-gc65be810-1jammy 2024-06-15T09:34:36.886 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:34:36.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-06-15T09:34:36.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-06-15T09:34:37.073 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-06-15T09:34:37.086 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-06-15T09:34:37.105 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-06-15T09:34:37.185 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy InRelease 2024-06-15T09:34:37.191 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-06-15T09:34:37.221 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-06-15T09:34:37.259 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy Release.gpg 2024-06-15T09:34:37.295 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2024-06-15T09:34:37.337 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-06-15T09:34:37.408 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 28.3 kB in 0s (86.0 kB/s) 2024-06-15T09:34:38.342 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:34:38.357 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2541-gc65be810-1jammy cephadm=19.0.0-2541-gc65be810-1jammy ceph-mds=19.0.0-2541-gc65be810-1jammy ceph-mgr=19.0.0-2541-gc65be810-1jammy ceph-common=19.0.0-2541-gc65be810-1jammy ceph-fuse=19.0.0-2541-gc65be810-1jammy ceph-test=19.0.0-2541-gc65be810-1jammy ceph-volume=19.0.0-2541-gc65be810-1jammy radosgw=19.0.0-2541-gc65be810-1jammy python3-rados=19.0.0-2541-gc65be810-1jammy python3-rgw=19.0.0-2541-gc65be810-1jammy python3-cephfs=19.0.0-2541-gc65be810-1jammy python3-rbd=19.0.0-2541-gc65be810-1jammy libcephfs2=19.0.0-2541-gc65be810-1jammy libcephfs-dev=19.0.0-2541-gc65be810-1jammy librados2=19.0.0-2541-gc65be810-1jammy librbd1=19.0.0-2541-gc65be810-1jammy rbd-fuse=19.0.0-2541-gc65be810-1jammy 2024-06-15T09:34:38.411 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-06-15T09:34:38.590 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-06-15T09:34:38.591 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-06-15T09:34:38.768 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-06-15T09:34:38.769 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-06-15T09:34:38.771 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-06-15T09:34:38.772 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet zip 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-06-15T09:34:38.773 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-06-15T09:34:38.774 INFO:teuthology.orchestra.run.smithi116.stdout: btrfs-tools 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-06-15T09:34:38.873 INFO:teuthology.orchestra.run.smithi116.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-06-15T09:34:38.875 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-06-15T09:34:38.876 INFO:teuthology.orchestra.run.smithi116.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-06-15T09:34:39.061 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 118 newly installed, 0 to remove and 313 not upgraded. 2024-06-15T09:34:39.061 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 180 MB of archives. 2024-06-15T09:34:39.061 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 822 MB of additional disk space will be used. 2024-06-15T09:34:39.061 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-06-15T09:34:39.090 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2541-gc65be810-1jammy [3,546 kB] 2024-06-15T09:34:39.483 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-06-15T09:34:39.495 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-06-15T09:34:39.576 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-06-15T09:34:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-06-15T09:34:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-06-15T09:34:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-06-15T09:34:39.855 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-06-15T09:34:39.855 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-06-15T09:34:39.857 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-06-15T09:34:39.875 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-06-15T09:34:39.879 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-06-15T09:34:39.884 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-06-15T09:34:39.887 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-06-15T09:34:39.968 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2541-gc65be810-1jammy [3,216 kB] 2024-06-15T09:34:39.968 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-06-15T09:34:39.972 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-06-15T09:34:39.976 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-06-15T09:34:39.980 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-06-15T09:34:39.981 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-06-15T09:34:39.981 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-06-15T09:34:39.983 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-06-15T09:34:39.984 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-06-15T09:34:39.985 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-06-15T09:34:40.051 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-06-15T09:34:40.067 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-06-15T09:34:40.068 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-06-15T09:34:40.068 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-06-15T09:34:40.074 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-06-15T09:34:40.074 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-06-15T09:34:40.075 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-06-15T09:34:40.076 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-06-15T09:34:40.078 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-06-15T09:34:40.132 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-06-15T09:34:40.161 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-06-15T09:34:40.163 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-06-15T09:34:40.164 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-06-15T09:34:40.165 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-06-15T09:34:40.176 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-06-15T09:34:40.176 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-06-15T09:34:40.178 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-06-15T09:34:40.178 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-06-15T09:34:40.214 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-06-15T09:34:40.263 INFO:teuthology.orchestra.run.smithi116.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-06-15T09:34:40.264 INFO:teuthology.orchestra.run.smithi116.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-06-15T09:34:40.267 INFO:teuthology.orchestra.run.smithi116.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-06-15T09:34:40.269 INFO:teuthology.orchestra.run.smithi116.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-06-15T09:34:40.273 INFO:teuthology.orchestra.run.smithi116.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-06-15T09:34:40.274 INFO:teuthology.orchestra.run.smithi116.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-06-15T09:34:40.276 INFO:teuthology.orchestra.run.smithi116.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-06-15T09:34:40.295 INFO:teuthology.orchestra.run.smithi116.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-06-15T09:34:40.327 INFO:teuthology.orchestra.run.smithi116.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2541-gc65be810-1jammy [711 kB] 2024-06-15T09:34:40.370 INFO:teuthology.orchestra.run.smithi116.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2541-gc65be810-1jammy [357 kB] 2024-06-15T09:34:40.404 INFO:teuthology.orchestra.run.smithi116.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2541-gc65be810-1jammy [32.8 kB] 2024-06-15T09:34:40.405 INFO:teuthology.orchestra.run.smithi116.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2541-gc65be810-1jammy [180 kB] 2024-06-15T09:34:40.432 INFO:teuthology.orchestra.run.smithi116.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-06-15T09:34:40.433 INFO:teuthology.orchestra.run.smithi116.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2541-gc65be810-1jammy [67.1 kB] 2024-06-15T09:34:40.433 INFO:teuthology.orchestra.run.smithi116.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-06-15T09:34:40.435 INFO:teuthology.orchestra.run.smithi116.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-06-15T09:34:40.436 INFO:teuthology.orchestra.run.smithi116.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2541-gc65be810-1jammy [330 kB] 2024-06-15T09:34:40.436 INFO:teuthology.orchestra.run.smithi116.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-06-15T09:34:40.438 INFO:teuthology.orchestra.run.smithi116.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-06-15T09:34:40.443 INFO:teuthology.orchestra.run.smithi116.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2541-gc65be810-1jammy [6,780 kB] 2024-06-15T09:34:40.486 INFO:teuthology.orchestra.run.smithi116.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-06-15T09:34:40.490 INFO:teuthology.orchestra.run.smithi116.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-06-15T09:34:40.490 INFO:teuthology.orchestra.run.smithi116.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-06-15T09:34:40.552 INFO:teuthology.orchestra.run.smithi116.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-06-15T09:34:40.553 INFO:teuthology.orchestra.run.smithi116.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-06-15T09:34:40.598 INFO:teuthology.orchestra.run.smithi116.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-06-15T09:34:40.599 INFO:teuthology.orchestra.run.smithi116.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-06-15T09:34:40.600 INFO:teuthology.orchestra.run.smithi116.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-06-15T09:34:40.601 INFO:teuthology.orchestra.run.smithi116.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-06-15T09:34:40.602 INFO:teuthology.orchestra.run.smithi116.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-06-15T09:34:40.603 INFO:teuthology.orchestra.run.smithi116.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-06-15T09:34:40.610 INFO:teuthology.orchestra.run.smithi116.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-06-15T09:34:40.679 INFO:teuthology.orchestra.run.smithi116.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-06-15T09:34:40.683 INFO:teuthology.orchestra.run.smithi116.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-06-15T09:34:40.684 INFO:teuthology.orchestra.run.smithi116.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-06-15T09:34:40.761 INFO:teuthology.orchestra.run.smithi116.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-06-15T09:34:40.768 INFO:teuthology.orchestra.run.smithi116.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-06-15T09:34:40.770 INFO:teuthology.orchestra.run.smithi116.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-06-15T09:34:40.771 INFO:teuthology.orchestra.run.smithi116.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-06-15T09:34:40.782 INFO:teuthology.orchestra.run.smithi116.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-06-15T09:34:40.786 INFO:teuthology.orchestra.run.smithi116.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-06-15T09:34:40.791 INFO:teuthology.orchestra.run.smithi116.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-06-15T09:34:40.839 INFO:teuthology.orchestra.run.smithi116.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2541-gc65be810-1jammy [112 kB] 2024-06-15T09:34:40.842 INFO:teuthology.orchestra.run.smithi116.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2541-gc65be810-1jammy [466 kB] 2024-06-15T09:34:40.842 INFO:teuthology.orchestra.run.smithi116.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-06-15T09:34:40.844 INFO:teuthology.orchestra.run.smithi116.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-06-15T09:34:40.844 INFO:teuthology.orchestra.run.smithi116.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-06-15T09:34:40.866 INFO:teuthology.orchestra.run.smithi116.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2541-gc65be810-1jammy [25.6 MB] 2024-06-15T09:34:40.924 INFO:teuthology.orchestra.run.smithi116.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-06-15T09:34:40.926 INFO:teuthology.orchestra.run.smithi116.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-06-15T09:34:40.944 INFO:teuthology.orchestra.run.smithi116.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-06-15T09:34:40.945 INFO:teuthology.orchestra.run.smithi116.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-06-15T09:34:40.945 INFO:teuthology.orchestra.run.smithi116.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-06-15T09:34:40.951 INFO:teuthology.orchestra.run.smithi116.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-06-15T09:34:40.952 INFO:teuthology.orchestra.run.smithi116.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-06-15T09:34:41.808 INFO:teuthology.orchestra.run.smithi116.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2541-gc65be810-1jammy [5,085 kB] 2024-06-15T09:34:41.896 INFO:teuthology.orchestra.run.smithi116.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2541-gc65be810-1jammy [244 kB] 2024-06-15T09:34:41.903 INFO:teuthology.orchestra.run.smithi116.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2541-gc65be810-1jammy [124 kB] 2024-06-15T09:34:41.906 INFO:teuthology.orchestra.run.smithi116.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2541-gc65be810-1jammy [1,526 kB] 2024-06-15T09:34:41.953 INFO:teuthology.orchestra.run.smithi116.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2541-gc65be810-1jammy [6,246 kB] 2024-06-15T09:34:42.161 INFO:teuthology.orchestra.run.smithi116.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2541-gc65be810-1jammy [22.7 MB] 2024-06-15T09:34:43.035 INFO:teuthology.orchestra.run.smithi116.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2541-gc65be810-1jammy [14.1 kB] 2024-06-15T09:34:43.035 INFO:teuthology.orchestra.run.smithi116.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2541-gc65be810-1jammy [869 kB] 2024-06-15T09:34:43.053 INFO:teuthology.orchestra.run.smithi116.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2541-gc65be810-1jammy [2,437 kB] 2024-06-15T09:34:43.097 INFO:teuthology.orchestra.run.smithi116.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2541-gc65be810-1jammy [791 kB] 2024-06-15T09:34:43.107 INFO:teuthology.orchestra.run.smithi116.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2541-gc65be810-1jammy [148 kB] 2024-06-15T09:34:43.108 INFO:teuthology.orchestra.run.smithi116.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2541-gc65be810-1jammy [2,331 kB] 2024-06-15T09:34:43.155 INFO:teuthology.orchestra.run.smithi116.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2541-gc65be810-1jammy [8,625 kB] 2024-06-15T09:34:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2541-gc65be810-1jammy [14.2 kB] 2024-06-15T09:34:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2541-gc65be810-1jammy [50.7 MB] 2024-06-15T09:34:45.361 INFO:teuthology.orchestra.run.smithi116.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2541-gc65be810-1jammy [121 kB] 2024-06-15T09:34:45.364 INFO:teuthology.orchestra.run.smithi116.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2541-gc65be810-1jammy [38.9 kB] 2024-06-15T09:34:45.365 INFO:teuthology.orchestra.run.smithi116.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2541-gc65be810-1jammy [13.4 MB] 2024-06-15T09:34:45.712 INFO:teuthology.orchestra.run.smithi116.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid-release/c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2541-gc65be810-1jammy [91.4 kB] 2024-06-15T09:34:46.280 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 180 MB in 7s (26.3 MB/s) 2024-06-15T09:34:46.443 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-06-15T09:34:46.541 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165982 files and directories currently installed.) 2024-06-15T09:34:46.546 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-06-15T09:34:46.625 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T09:34:46.945 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-06-15T09:34:46.971 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-06-15T09:34:47.003 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T09:34:47.332 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-06-15T09:34:47.358 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-06-15T09:34:47.398 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T09:34:47.843 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-06-15T09:34:47.870 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T09:34:47.918 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:34:48.280 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-06-15T09:34:48.306 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T09:34:48.338 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:34:48.691 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-06-15T09:34:48.717 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-06-15T09:34:48.758 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:34:49.119 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-06-15T09:34:49.145 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-06-15T09:34:49.185 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T09:34:49.455 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librados2. 2024-06-15T09:34:49.470 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../007-librados2_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:49.505 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librados2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:49.916 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librbd1. 2024-06-15T09:34:49.942 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../008-librbd1_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:49.982 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librbd1 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:50.444 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs2. 2024-06-15T09:34:50.470 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:50.510 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:50.754 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rados. 2024-06-15T09:34:50.769 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:50.804 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rados (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:51.124 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-06-15T09:34:51.150 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:34:51.190 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-argparse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:51.426 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cephfs. 2024-06-15T09:34:51.441 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:51.484 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cephfs (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:51.778 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-common. 2024-06-15T09:34:51.805 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:34:51.845 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-common (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:52.123 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-wcwidth. 2024-06-15T09:34:52.149 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-06-15T09:34:52.181 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T09:34:52.492 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-prettytable. 2024-06-15T09:34:52.519 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-06-15T09:34:52.559 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-06-15T09:34:52.870 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rbd. 2024-06-15T09:34:52.896 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:52.944 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rbd (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:53.298 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-06-15T09:34:53.324 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-06-15T09:34:53.364 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T09:34:53.608 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-06-15T09:34:53.624 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-06-15T09:34:53.658 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-06-15T09:34:53.969 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-06-15T09:34:53.995 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-06-15T09:34:54.036 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-15T09:34:54.364 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua5.1. 2024-06-15T09:34:54.390 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-06-15T09:34:54.431 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-06-15T09:34:54.742 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-any. 2024-06-15T09:34:54.757 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-06-15T09:34:54.792 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-any (27ubuntu1) ... 2024-06-15T09:34:54.986 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package zip. 2024-06-15T09:34:55.001 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-06-15T09:34:55.044 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking zip (3.0-12build2) ... 2024-06-15T09:34:55.364 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package unzip. 2024-06-15T09:34:55.390 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-06-15T09:34:55.431 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-06-15T09:34:55.826 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package luarocks. 2024-06-15T09:34:55.852 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-06-15T09:34:55.893 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-06-15T09:34:56.300 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librgw2. 2024-06-15T09:34:56.326 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../025-librgw2_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:56.366 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librgw2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:56.877 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rgw. 2024-06-15T09:34:56.892 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:56.927 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rgw (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:57.272 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liboath0:amd64. 2024-06-15T09:34:57.298 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-06-15T09:34:57.338 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T09:34:57.958 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libradosstriper1. 2024-06-15T09:34:57.984 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:58.025 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libradosstriper1 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:58.378 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-common. 2024-06-15T09:34:58.404 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:58.436 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-common (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:34:59.825 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-base. 2024-06-15T09:34:59.851 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:34:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-base (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:00.520 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-06-15T09:35:00.546 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-06-15T09:35:00.586 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-06-15T09:35:00.856 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cheroot. 2024-06-15T09:35:00.871 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-06-15T09:35:00.906 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T09:35:01.218 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-06-15T09:35:01.244 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-06-15T09:35:01.284 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-06-15T09:35:01.587 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.text. 2024-06-15T09:35:01.613 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-06-15T09:35:01.653 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-06-15T09:35:01.941 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-06-15T09:35:01.960 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-06-15T09:35:01.991 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-06-15T09:35:02.286 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempora. 2024-06-15T09:35:02.312 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-06-15T09:35:02.352 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-06-15T09:35:02.613 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-portend. 2024-06-15T09:35:02.640 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-06-15T09:35:02.673 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-06-15T09:35:03.175 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-06-15T09:35:03.201 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-06-15T09:35:03.241 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-06-15T09:35:03.544 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cherrypy3. 2024-06-15T09:35:03.570 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-06-15T09:35:03.611 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-06-15T09:35:03.922 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-natsort. 2024-06-15T09:35:03.938 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-06-15T09:35:03.972 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-06-15T09:35:04.275 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-logutils. 2024-06-15T09:35:04.302 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-06-15T09:35:04.342 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-06-15T09:35:04.670 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-markupsafe. 2024-06-15T09:35:04.696 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-06-15T09:35:04.736 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-06-15T09:35:05.022 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-mako. 2024-06-15T09:35:05.038 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-06-15T09:35:05.080 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T09:35:05.367 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplegeneric. 2024-06-15T09:35:05.382 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-06-15T09:35:05.417 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-06-15T09:35:05.686 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-singledispatch. 2024-06-15T09:35:05.713 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-06-15T09:35:05.753 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-06-15T09:35:06.047 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webob. 2024-06-15T09:35:06.073 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-06-15T09:35:06.114 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-06-15T09:35:06.567 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-waitress. 2024-06-15T09:35:06.593 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-06-15T09:35:06.637 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T09:35:06.936 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempita. 2024-06-15T09:35:06.963 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-06-15T09:35:07.003 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T09:35:07.264 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-paste. 2024-06-15T09:35:07.290 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-06-15T09:35:07.322 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T09:35:07.617 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-06-15T09:35:07.632 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-06-15T09:35:07.667 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T09:35:07.903 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastedeploy. 2024-06-15T09:35:07.929 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-06-15T09:35:07.969 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-06-15T09:35:08.306 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webtest. 2024-06-15T09:35:08.332 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-06-15T09:35:08.372 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-06-15T09:35:08.700 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pecan. 2024-06-15T09:35:08.727 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-06-15T09:35:08.767 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T09:35:09.121 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-certifi. 2024-06-15T09:35:09.137 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-06-15T09:35:09.179 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-06-15T09:35:09.508 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-idna. 2024-06-15T09:35:09.534 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-06-15T09:35:09.574 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T09:35:09.860 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-urllib3. 2024-06-15T09:35:09.887 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-06-15T09:35:09.926 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T09:35:10.205 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests. 2024-06-15T09:35:10.231 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-06-15T09:35:10.271 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T09:35:10.607 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-werkzeug. 2024-06-15T09:35:10.634 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-06-15T09:35:10.674 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T09:35:10.969 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-06-15T09:35:10.984 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:11.027 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:11.407 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-06-15T09:35:11.433 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:11.473 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:11.759 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-bcrypt. 2024-06-15T09:35:11.786 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-06-15T09:35:11.826 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-06-15T09:35:12.095 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-openssl. 2024-06-15T09:35:12.122 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-06-15T09:35:12.162 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-06-15T09:35:13.274 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr. 2024-06-15T09:35:13.301 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:13.340 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:13.735 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mon. 2024-06-15T09:35:13.762 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:13.801 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mon (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:14.313 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-osd. 2024-06-15T09:35:14.328 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:14.363 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-osd (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:15.449 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph. 2024-06-15T09:35:15.464 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../066-ceph_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:15.516 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:15.835 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-fuse. 2024-06-15T09:35:15.862 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:15.901 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-fuse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:16.238 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mds. 2024-06-15T09:35:16.265 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:16.296 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mds (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:16.674 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephadm. 2024-06-15T09:35:16.701 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../069-cephadm_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:16.740 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephadm (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:17.052 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-asyncssh. 2024-06-15T09:35:17.078 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-06-15T09:35:17.118 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-06-15T09:35:17.446 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-babel-localedata. 2024-06-15T09:35:17.473 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T09:35:17.512 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T09:35:18.710 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-babel. 2024-06-15T09:35:18.737 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-06-15T09:35:18.776 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T09:35:19.120 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jinja2. 2024-06-15T09:35:19.147 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-06-15T09:35:19.187 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T09:35:19.523 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-06-15T09:35:19.538 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:19.918 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-repoze.lru. 2024-06-15T09:35:19.944 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-06-15T09:35:19.984 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-06-15T09:35:20.287 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-routes. 2024-06-15T09:35:20.314 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-06-15T09:35:20.353 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T09:35:20.707 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-06-15T09:35:20.734 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:20.773 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:22.003 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-06-15T09:35:22.019 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-06-15T09:35:22.053 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T09:35:22.565 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-joblib. 2024-06-15T09:35:22.592 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-06-15T09:35:22.631 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T09:35:22.993 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-06-15T09:35:23.020 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-06-15T09:35:23.059 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-06-15T09:35:23.362 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn. 2024-06-15T09:35:23.389 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-06-15T09:35:23.428 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T09:35:24.104 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-06-15T09:35:24.132 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:24.256 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:25.184 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cachetools. 2024-06-15T09:35:25.200 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-06-15T09:35:25.234 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-06-15T09:35:25.520 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1. 2024-06-15T09:35:25.548 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-06-15T09:35:25.590 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-06-15T09:35:25.932 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-06-15T09:35:25.959 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-06-15T09:35:25.998 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T09:35:26.310 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rsa. 2024-06-15T09:35:26.337 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-06-15T09:35:26.376 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rsa (4.8-1) ... 2024-06-15T09:35:26.663 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-google-auth. 2024-06-15T09:35:26.690 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-06-15T09:35:26.729 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-06-15T09:35:27.049 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-06-15T09:35:27.076 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-06-15T09:35:27.116 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T09:35:27.437 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-websocket. 2024-06-15T09:35:27.463 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-06-15T09:35:27.502 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-06-15T09:35:27.822 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-kubernetes. 2024-06-15T09:35:27.849 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-06-15T09:35:27.888 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T09:35:28.561 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-06-15T09:35:28.577 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:28.610 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:29.138 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libonig5:amd64. 2024-06-15T09:35:29.166 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-06-15T09:35:29.204 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T09:35:29.549 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjq1:amd64. 2024-06-15T09:35:29.577 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T09:35:29.616 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T09:35:29.877 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package jq. 2024-06-15T09:35:29.892 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-06-15T09:35:29.927 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-06-15T09:35:30.146 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package socat. 2024-06-15T09:35:30.173 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-06-15T09:35:30.212 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-06-15T09:35:30.524 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package xmlstarlet. 2024-06-15T09:35:30.540 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-06-15T09:35:30.582 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-06-15T09:35:30.936 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-test. 2024-06-15T09:35:30.953 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:30.994 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-test (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:33.240 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-volume. 2024-06-15T09:35:33.268 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2541-gc65be810-1jammy_all.deb ... 2024-06-15T09:35:33.323 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-volume (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:33.602 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs-dev. 2024-06-15T09:35:33.618 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:33.652 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs-dev (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:33.938 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-socket:amd64. 2024-06-15T09:35:33.966 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-06-15T09:35:34.005 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-15T09:35:34.610 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-sec:amd64. 2024-06-15T09:35:34.637 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-06-15T09:35:34.676 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-06-15T09:35:35.021 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package nvme-cli. 2024-06-15T09:35:35.049 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-06-15T09:35:35.087 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T09:35:35.476 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pkg-config. 2024-06-15T09:35:35.503 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-06-15T09:35:35.534 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-06-15T09:35:35.745 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-06-15T09:35:35.761 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-06-15T09:35:35.803 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-06-15T09:35:36.240 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-iniconfig. 2024-06-15T09:35:36.267 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-06-15T09:35:36.298 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-06-15T09:35:36.593 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastescript. 2024-06-15T09:35:36.620 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-06-15T09:35:36.659 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-06-15T09:35:37.021 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pluggy. 2024-06-15T09:35:37.048 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-06-15T09:35:37.087 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-06-15T09:35:37.390 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-psutil. 2024-06-15T09:35:37.417 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-06-15T09:35:37.456 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-06-15T09:35:37.793 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-py. 2024-06-15T09:35:37.820 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-06-15T09:35:37.859 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-py (1.10.0-1) ... 2024-06-15T09:35:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pygments. 2024-06-15T09:35:38.206 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-06-15T09:35:38.245 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T09:35:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyinotify. 2024-06-15T09:35:38.760 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-06-15T09:35:38.799 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-06-15T09:35:39.119 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-toml. 2024-06-15T09:35:39.146 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-06-15T09:35:39.277 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-06-15T09:35:39.530 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pytest. 2024-06-15T09:35:39.557 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-06-15T09:35:39.587 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T09:35:39.933 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplejson. 2024-06-15T09:35:39.960 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-06-15T09:35:39.999 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-06-15T09:35:40.260 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-06-15T09:35:40.276 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-06-15T09:35:40.310 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-06-15T09:35:40.964 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package radosgw. 2024-06-15T09:35:40.992 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../116-radosgw_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:41.030 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking radosgw (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:41.841 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package rbd-fuse. 2024-06-15T09:35:41.858 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2541-gc65be810-1jammy_amd64.deb ... 2024-06-15T09:35:41.900 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking rbd-fuse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:42.210 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-06-15T09:35:42.473 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-06-15T09:35:42.582 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-06-15T09:35:42.832 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-06-15T09:35:43.133 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-06-15T09:35:43.571 INFO:teuthology.orchestra.run.smithi116.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-06-15T09:35:43.626 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephadm (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:44.200 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user cephadm....done 2024-06-15T09:35:44.340 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-06-15T09:35:44.608 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-06-15T09:35:44.851 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-06-15T09:35:44.977 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-06-15T09:35:45.245 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-06-15T09:35:45.504 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-06-15T09:35:45.630 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-py (1.10.0-1) ... 2024-06-15T09:35:45.932 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-06-15T09:35:46.283 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-06-15T09:35:46.576 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-06-15T09:35:46.844 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-06-15T09:35:47.104 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-06-15T09:35:47.355 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-06-15T09:35:47.639 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-06-15T09:35:47.874 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-argparse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:48.158 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-06-15T09:35:48.259 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-06-15T09:35:48.385 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-06-15T09:35:48.511 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-06-15T09:35:48.771 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-06-15T09:35:49.152 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-06-15T09:35:49.186 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-06-15T09:35:49.241 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-06-15T09:35:49.342 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-06-15T09:35:49.944 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-06-15T09:35:50.110 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-06-15T09:35:50.387 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-06-15T09:35:50.638 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-06-15T09:35:50.947 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up zip (3.0-12build2) ... 2024-06-15T09:35:51.057 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-06-15T09:35:51.717 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-06-15T09:35:51.952 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-06-15T09:35:52.086 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-06-15T09:35:52.212 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-06-15T09:35:52.505 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-06-15T09:35:52.898 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-06-15T09:35:53.140 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-06-15T09:35:53.500 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-06-15T09:35:53.793 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-06-15T09:35:54.136 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-06-15T09:35:54.387 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-06-15T09:35:54.513 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-common (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:35:54.823 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-06-15T09:35:55.124 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-06-15T09:35:56.313 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-06-15T09:35:56.556 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:35:56.683 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-toml (0.10.2-1) ... 2024-06-15T09:35:56.934 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-06-15T09:35:57.060 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-06-15T09:35:57.186 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-06-15T09:35:57.479 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-06-15T09:35:57.738 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-06-15T09:35:57.989 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:35:58.107 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-06-15T09:35:58.417 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-06-15T09:35:58.684 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-06-15T09:35:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-06-15T09:35:59.195 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-06-15T09:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-06-15T09:35:59.714 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-06-15T09:35:59.840 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-06-15T09:36:00.124 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-06-15T09:36:00.234 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-06-15T09:36:00.510 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-06-15T09:36:00.811 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-06-15T09:36:01.062 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-06-15T09:36:01.188 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-06-15T09:36:01.314 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-06-15T09:36:01.403 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-06-15T09:36:01.649 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-06-15T09:36:01.775 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-06-15T09:36:02.160 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-06-15T09:36:02.420 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-any (27ubuntu1) ... 2024-06-15T09:36:02.546 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-portend (3.0.0-1) ... 2024-06-15T09:36:02.822 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-06-15T09:36:02.948 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-06-15T09:36:03.274 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-06-15T09:36:03.575 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-06-15T09:36:03.826 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-06-15T09:36:03.961 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rsa (4.8-1) ... 2024-06-15T09:36:04.237 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-06-15T09:36:04.505 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-06-15T09:36:04.866 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-06-15T09:36:05.243 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-06-15T09:36:05.719 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-06-15T09:36:06.047 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-06-15T09:36:06.165 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librados2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:06.273 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:06.383 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-06-15T09:36:06.719 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:06.845 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libradosstriper1 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:06.971 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-06-15T09:36:07.247 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librbd1 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:07.406 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-modules-core (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:07.491 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-fuse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:07.691 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T09:36:07.692 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-06-15T09:36:08.200 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs-dev (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:08.327 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rados (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:08.461 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librgw2 (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:08.587 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-06-15T09:36:09.799 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rbd (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:09.925 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up rbd-fuse (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:10.051 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rgw (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:10.177 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cephfs (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:10.303 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-common (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:10.854 INFO:teuthology.orchestra.run.smithi116.stdout:Adding group ceph....done 2024-06-15T09:36:11.172 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user ceph....done 2024-06-15T09:36:11.366 INFO:teuthology.orchestra.run.smithi116.stdout:Setting system user ceph properties....done 2024-06-15T09:36:11.385 INFO:teuthology.orchestra.run.smithi116.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-06-15T09:36:11.522 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-06-15T09:36:11.854 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-06-15T09:36:12.355 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-test (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:12.498 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up radosgw (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:13.334 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T09:36:13.334 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-06-15T09:36:13.800 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-base (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:14.152 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-06-15T09:36:14.610 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mds (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:14.817 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T09:36:14.817 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-06-15T09:36:15.279 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:15.497 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T09:36:15.497 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-06-15T09:36:16.242 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-osd (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:16.536 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T09:36:16.536 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-06-15T09:36:16.986 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:17.104 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:17.271 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mon (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:17.488 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T09:36:17.489 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-06-15T09:36:17.931 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-cephadm (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:18.073 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:18.199 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-dashboard (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:18.350 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-volume (19.0.0-2541-gc65be810-1jammy) ... 2024-06-15T09:36:18.686 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-06-15T09:36:18.779 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-06-15T09:36:18.923 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-06-15T09:36:20.040 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-06-15T09:36:20.713 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-06-15T09:36:20.718 DEBUG:teuthology.parallel:result is None 2024-06-15T09:36:20.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c65be8102fd0c62c21484f7d2c7085bf8cb6a8fd 2024-06-15T09:36:20.944 DEBUG:teuthology.orchestra.run.smithi116:> dpkg-query -W -f '${Version}' ceph 2024-06-15T09:36:20.963 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2541-gc65be810-1jammy 2024-06-15T09:36:20.963 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2541-gc65be810-1jammy 2024-06-15T09:36:20.964 INFO:teuthology.task.install:The correct ceph version 19.0.0-2541-gc65be810-1jammy is installed. 2024-06-15T09:36:20.965 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-15T09:36:20.966 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:36:20.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-15T09:36:21.018 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-15T09:36:21.018 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:36:21.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-06-15T09:36:21.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-15T09:36:21.129 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-15T09:36:21.129 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:36:21.129 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-15T09:36:21.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-15T09:36:21.241 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-15T09:36:21.241 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:36:21.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-06-15T09:36:21.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-15T09:36:21.353 INFO:teuthology.run_tasks:Running task workunit... 2024-06-15T09:36:21.367 INFO:tasks.workunit:Pulling workunits from ref db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-06-15T09:36:21.368 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-15T09:36:21.368 INFO:tasks.workunit:timeout=3h 2024-06-15T09:36:21.368 INFO:tasks.workunit:cleanup=True 2024-06-15T09:36:21.368 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-15T09:36:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T09:36:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-15T09:36:21.402 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-15T09:36:21.450 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-15T09:36:21.450 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-15T09:36:21.498 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-06-15T09:36:21.548 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'db0330b1e4e2470d52b750e251e55a522b4f7d69'. 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-06-15T09:37:06.492 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-06-15T09:37:06.493 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at db0330b1e4e Merge pull request #56234 from zdover23/wip-doc-2024-03-16-backport-56182-to-squid 2024-06-15T09:37:06.498 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-15T09:37:06.546 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-06-15T09:37:06.547 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-15T09:37:06.593 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-06-15T09:37:06.594 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-06-15T09:37:06.594 DEBUG:teuthology.orchestra.run.smithi116:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-06-15T09:37:06.660 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-15T09:37:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-06-15T09:37:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none ' 2024-06-15T09:37:06.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-06-15T09:37:06.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-15T09:37:06.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-06-15T09:37:06.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-15T09:37:06.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:37:06.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:37:06.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:37:06.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:37:06.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:37:06.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:37:06.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:37:06.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:37:06.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:37:06.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:37:06.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:37:06.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:37:06.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:37:06.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:37:06.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:37:06.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-15T09:37:06.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:37:06.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:37:06.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:37:06.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20638 2024-06-15T09:37:06.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:37:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:37:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-06-15T09:37:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:37:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:37:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:37:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20638 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-06-15T09:37:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-06-15T09:37:06.684 INFO:tasks.workunit.client.0.smithi116.stdout:Dir: td/osd-mapper 2024-06-15T09:37:06.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-06-15T09:37:06.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-15T09:37:06.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-15T09:37:08.625 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 971fd802-e5b4-43cf-a728-c0cc4d756d19 2024-06-15T09:37:08.897 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:37:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.002+0000 7fb5fd5dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:09.008 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.006+0000 7fb5fd5dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.010+0000 7fb5fd5dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:09.010 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.010+0000 7fb5fd5dd7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-15T09:37:09.010 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.010+0000 7fb5fd5dd7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-15T09:37:09.010 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.010+0000 7fb5fd5dd7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-06-15T09:37:09.040 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:09.038+0000 7fb5fd5dd7c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-06-15T09:37:15.135 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:37:15.334 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:37:15.370 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:15.370+0000 7f7dfb19f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:15.398 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:15.398+0000 7f7dfb19f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:15.399 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:15.398+0000 7f7dfb19f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:15.558 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:37:15.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:15.966+0000 7f7dfb19f7c0 -1 Falling back to public interface 2024-06-15T09:37:16.901 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:37:18.142 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:37:18.549 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:18.550+0000 7f7dfb19f7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:37:19.376 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:37:20.727 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:37:21.965 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:37:23.229 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:37:24.531 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:37:25.769 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:37:27.002 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:37:28.243 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:37:29.518 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:37:30.758 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:37:31.994 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:37:33.622 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:37:34.982 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:37:36.654 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:37:37.894 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:37:39.236 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:37:40.472 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:37:41.838 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:37:43.436 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:37:45.002 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:37:45.266 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1561932359,v1:127.0.0.1:6801/1561932359] [v2:127.0.0.1:6802/1561932359,v1:127.0.0.1:6803/1561932359] exists,up 971fd802-e5b4-43cf-a728-c0cc4d756d19 2024-06-15T09:37:45.271 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 925c8271-eadd-4aae-8273-082413e222d2 2024-06-15T09:37:45.597 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:37:45.655 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.654+0000 7f615207c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.698+0000 7f615207c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.702+0000 7f615207c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.702+0000 7f615207c7c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-15T09:37:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.702+0000 7f615207c7c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-15T09:37:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.702+0000 7f615207c7c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-06-15T09:37:45.769 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:45.770+0000 7f615207c7c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-06-15T09:37:52.226 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:37:52.492 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:37:52.527 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:52.526+0000 7fbae00ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:52.529 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:52.530+0000 7fbae00ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:52.530 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:52.530+0000 7fbae00ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:37:52.768 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:37:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:53.078+0000 7fbae00ec7c0 -1 Falling back to public interface 2024-06-15T09:37:54.006 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:37:55.252 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:37:55.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:37:55.758+0000 7fbae00ec7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:37:56.492 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:37:57.893 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:37:59.132 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:38:00.420 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:38:01.706 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:38:02.937 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:38:04.170 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:38:05.427 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:38:06.732 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:38:08.088 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:38:09.330 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:38:10.878 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:38:12.652 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:38:14.520 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:38:16.420 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:38:18.075 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:38:19.710 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:38:21.253 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:38:21.490 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2577808824,v1:127.0.0.1:6811/2577808824] [v2:127.0.0.1:6812/2577808824,v1:127.0.0.1:6813/2577808824] exists,up 925c8271-eadd-4aae-8273-082413e222d2 2024-06-15T09:38:21.496 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 45715acd-a9e2-43e8-bc50-20d68b10f58a 2024-06-15T09:38:21.868 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:38:21.924 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.921+0000 7f53bf9fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:21.970 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.969+0000 7f53bf9fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:21.971 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.969+0000 7f53bf9fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:21.972 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.969+0000 7f53bf9fa7c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-15T09:38:21.972 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.969+0000 7f53bf9fa7c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-15T09:38:21.972 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:21.969+0000 7f53bf9fa7c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-06-15T09:38:22.053 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:22.053+0000 7f53bf9fa7c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-06-15T09:38:28.612 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:38:28.886 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:38:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:28.921+0000 7f77539247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:28.952 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:28.949+0000 7f77539247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:28.953 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:28.953+0000 7f77539247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:38:29.153 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:38:29.543 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:29.541+0000 7f77539247c0 -1 Falling back to public interface 2024-06-15T09:38:30.416 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:38:31.659 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:38:32.258 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:38:32.257+0000 7f77539247c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:38:32.896 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:38:34.165 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:38:35.408 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:38:36.649 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:38:38.075 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:38:39.312 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:38:40.551 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:38:41.796 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:38:43.143 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:38:44.466 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:38:45.771 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:38:47.200 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:38:48.653 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:38:50.334 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:38:51.992 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:38:53.540 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:38:55.280 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:38:56.666 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:38:57.905 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:38:58.141 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3656217483,v1:127.0.0.1:6819/3656217483] [v2:127.0.0.1:6820/3656217483,v1:127.0.0.1:6821/3656217483] exists,up 45715acd-a9e2-43e8-bc50-20d68b10f58a 2024-06-15T09:38:58.502 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' created 2024-06-15T09:39:00.626 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836501 2024-06-15T09:39:00.893 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672974 2024-06-15T09:39:01.263 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509447 2024-06-15T09:39:04.732 INFO:tasks.workunit.client.0.smithi116.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-06-15T09:39:04.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-06-15T09:39:04.852 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: { 2024-06-15T09:39:04.852 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:04.852 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:04.865 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: { 2024-06-15T09:39:04.866 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:04.866 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:04.879 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2: { 2024-06-15T09:39:04.879 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:04.879 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:04.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-06-15T09:39:05.011 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: { 2024-06-15T09:39:05.011 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:05.011 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:05.023 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: { 2024-06-15T09:39:05.023 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:05.023 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:05.035 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2: { 2024-06-15T09:39:05.035 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-06-15T09:39:05.035 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-06-15T09:39:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-06-15T09:39:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-06-15T09:39:05.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-06-15T09:39:05.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-06-15T09:39:05.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-15T09:39:05.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-15T09:39:05.383 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap01 2024-06-15T09:39:05.598 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap02 2024-06-15T09:39:05.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-06-15T09:39:05.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-15T09:39:05.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-15T09:39:06.087 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap13 2024-06-15T09:39:06.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-06-15T09:39:06.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-06-15T09:39:06.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-06-15T09:39:06.611 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap24 2024-06-15T09:39:06.904 INFO:tasks.workunit.client.0.smithi116.stdout:created pool test snap snap25 2024-06-15T09:39:06.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 11970 2024-06-15T09:39:06.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-06-15T09:39:07.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-15T09:39:07.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-06-15T09:39:07.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-06-15T09:39:07.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-15T09:39:07.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-15T09:39:07.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-06-15T09:39:07.771 INFO:tasks.workunit.client.0.smithi116.stdout:pgid is 1.3 (primary: osd.1) 2024-06-15T09:39:07.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-06-15T09:39:07.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-06-15T09:39:07.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-06-15T09:39:07.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-06-15T09:39:07.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-06-15T09:39:08.003 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs_brief 2024-06-15T09:39:08.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-06-15T09:39:08.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-06-15T09:39:08.020 INFO:tasks.workunit.client.0.smithi116.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:39:08.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20638/ceph-osd.1.asok 2024-06-15T09:39:08.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-06-15T09:39:08.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.20638/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-06-15T09:39:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-15T09:39:08.184 INFO:tasks.workunit.client.0.smithi116.stdout:{"success":true}{ 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "objxxx", 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "clones": [ 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap01" 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2024-06-15T09:39:08.185 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap02" 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 3, 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 3, 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap13" 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:08.186 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 5, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 4, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap24" 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 5, 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap25" 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:08.187 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: "id": "head", 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [], 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:08.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-15T09:39:08.311 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:08.311 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:39:08.311 INFO:tasks.workunit.client.0.smithi116.stdout: "must": true, 2024-06-15T09:39:08.311 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "0.000000" 2024-06-15T09:39:08.311 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-06-15T09:39:11.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-15T09:39:11.545 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 3 0 0 0 0 18 0 0 5 0 5 active+clean 2024-06-15T09:39:00.013109+0000 24'5 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:00.013534+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:38:59.593307+0000 0'0 26:31 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:00.012288+0000 0'0 24:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:39:11.546 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:39:11.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-06-15T09:39:11.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 3 0 0 0 0 18 0 0 5 0 5 active+clean 2024-06-15T09:39:00.013109+0000 24'5 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:00.013534+0000 0'0 26:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:38:59.593307+0000 0'0 26:31 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:00.012288+0000 0'0 24:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:39:11.824 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:39:11.825 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:39:11.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-06-15T09:39:12.381 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T09:39:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-06-15T09:39:12.762 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T09:39:12.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-06-15T09:39:17.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-06-15T09:39:17.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-15T09:39:17.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:39:17.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:39:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:39:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:39:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:39:18.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:39:18.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:18.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-15T09:39:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.129+0000 7f83f8ed1ac0 1 bluefs mount 2024-06-15T09:39:18.141 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.141+0000 7f83f8ed1ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:18.171 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.169+0000 7f83f8ed1ac0 1 bluefs mount shared_bdev_used = 0 2024-06-15T09:39:18.172 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.169+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:18.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.173+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-15T09:39:18.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 freelist init 2024-06-15T09:39:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 freelist _read_cfg 2024-06-15T09:39:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-06-15T09:39:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bluefs umount 2024-06-15T09:39:18.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.225+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8700 td/osd-mapper/0/block) close 2024-06-15T09:39:18.408 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.409+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.409+0000 7f83f8ed1ac0 1 bdev(0x55a9a1ec8700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.409+0000 7f83f8ed1ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-15T09:39:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.409+0000 7f83f8ed1ac0 1 bluefs mount 2024-06-15T09:39:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.409+0000 7f83f8ed1ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.413+0000 7f83f8ed1ac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-15T09:39:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.413+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:18.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:18.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:18.417+0000 7f83f8ed1ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.577+0000 7f83f8ed1ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:19.582 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-15T09:39:19.582 INFO:tasks.workunit.client.0.smithi116.stdout:00000000 0b 05 58 02 00 00 c8 8f 3c 4c c7 7a 42 91 a0 20 |..X.....= 2 )) 2024-06-15T09:39:19.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-15T09:39:19.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bdev(0x55f177dec000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bdev(0x55f177dec000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-15T09:39:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.945+0000 7fad03675ac0 1 bluefs mount 2024-06-15T09:39:19.954 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.953+0000 7fad03675ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:19.963 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.961+0000 7fad03675ac0 1 bluefs mount shared_bdev_used = 0 2024-06-15T09:39:19.963 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.961+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:19.966 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:19.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:19.965+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:20.014 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 freelist init 2024-06-15T09:39:20.015 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 freelist _read_cfg 2024-06-15T09:39:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-06-15T09:39:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bluefs umount 2024-06-15T09:39:20.016 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.013+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) close 2024-06-15T09:39:20.228 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.229+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:20.229 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.229+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:20.229 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.229+0000 7fad03675ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-15T09:39:20.229 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.229+0000 7fad03675ac0 1 bluefs mount 2024-06-15T09:39:20.229 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.229+0000 7fad03675ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:20.234 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.233+0000 7fad03675ac0 1 bluefs mount shared_bdev_used = 27459584 2024-06-15T09:39:20.234 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.233+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:20.237 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:20.237 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:20.237 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:20.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:20.237+0000 7fad03675ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:21.249 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:21.249+0000 7fad03675ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:21.251 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:21.249+0000 7fad03675ac0 1 bluefs umount 2024-06-15T09:39:21.252 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:21.249+0000 7fad03675ac0 1 bdev(0x55f177dec700 td/osd-mapper/2/block) close 2024-06-15T09:39:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:21.497+0000 7fad03675ac0 1 freelist shutdown 2024-06-15T09:39:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:21.497+0000 7fad03675ac0 1 bdev(0x55f177dec000 td/osd-mapper/2/block) close 2024-06-15T09:39:21.736 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-15T09:39:21.736 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-15T09:39:21.736 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:21.737 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-15T09:39:21.737 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-15T09:39:21.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:21.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-06-15T09:39:21.738 INFO:tasks.workunit.client.0.smithi116.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-06-15T09:39:21.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:21.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-15T09:39:22.387 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-15T09:39:22.487 INFO:tasks.workunit.client.0.smithi116.stdout:00000000 0b 05 58 02 00 00 c8 8f 3c 4c c7 7a 42 91 a0 20 |..X.....= 3 )) 2024-06-15T09:39:24.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-06-15T09:39:24.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bdev(0x5635c1434000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:24.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bdev(0x5635c1434000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:24.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-15T09:39:24.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bdev(0x5635c1434700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:24.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bdev(0x5635c1434700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:24.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-15T09:39:24.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.429+0000 7f9bc07b5ac0 1 bluefs mount 2024-06-15T09:39:24.440 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.441+0000 7f9bc07b5ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:24.449 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.449+0000 7f9bc07b5ac0 1 bluefs mount shared_bdev_used = 0 2024-06-15T09:39:24.449 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.449+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.449+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:24.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.453+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-06-15T09:39:24.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 freelist init 2024-06-15T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 freelist _read_cfg 2024-06-15T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-06-15T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bluefs umount 2024-06-15T09:39:24.459 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.457+0000 7f9bc07b5ac0 1 bdev(0x5635c1434700 td/osd-mapper/0/block) close 2024-06-15T09:39:24.712 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.713+0000 7f9bc07b5ac0 1 bdev(0x5635c1434700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-06-15T09:39:24.712 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.713+0000 7f9bc07b5ac0 1 bdev(0x5635c1434700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:24.712 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.713+0000 7f9bc07b5ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-06-15T09:39:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.713+0000 7f9bc07b5ac0 1 bluefs mount 2024-06-15T09:39:24.731 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.729+0000 7f9bc07b5ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:24.763 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.761+0000 7f9bc07b5ac0 1 bluefs mount shared_bdev_used = 9568256 2024-06-15T09:39:24.763 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.761+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:24.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:24.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:24.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:24.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:24.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:24.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:24.765+0000 7f9bc07b5ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:25.064 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:25.065+0000 7f9bc07b5ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:25.067 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-15T09:39:25.067 INFO:tasks.workunit.client.0.smithi116.stdout:00000000 0b 05 58 02 00 00 c8 8f 3c 4c c7 7a 42 91 a0 20 |..X.....= 1 )) 2024-06-15T09:39:26.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-15T09:39:26.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-15T09:39:26.543 INFO:tasks.workunit.client.0.smithi116.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:26.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-15T09:39:26.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.qljagHOvnO_the_val 2024-06-15T09:39:26.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:26.546 INFO:tasks.workunit.client.0.smithi116.stdout:Value dumped in: /tmp/tmp.qljagHOvnO_the_val 2024-06-15T09:39:26.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.qljagHOvnO_the_val' 2024-06-15T09:39:26.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.qljagHOvnO_the_val 2024-06-15T09:39:27.167 INFO:tasks.workunit.client.0.smithi116.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-15T09:39:27.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:27.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.qljagHOvnO_the_val 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-15T09:39:27.628 INFO:tasks.workunit.client.0.smithi116.stdout:0000073 2024-06-15T09:39:27.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-15T09:39:27.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:28.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.qljagHOvnO_the_val 2024-06-15T09:39:29.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:29.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:29.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-15T09:39:29.770 INFO:tasks.workunit.client.0.smithi116.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-06-15T09:39:29.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-06-15T09:39:29.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-15T09:39:29.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-15T09:39:29.798 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bdev(0x55f5facec000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bdev(0x55f5facec000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bdev(0x55f5facec700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bdev(0x55f5facec700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-15T09:39:29.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.797+0000 7f68273deac0 1 bluefs mount 2024-06-15T09:39:29.802 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.801+0000 7f68273deac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:29.811 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.809+0000 7f68273deac0 1 bluefs mount shared_bdev_used = 0 2024-06-15T09:39:29.812 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.809+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:29.816 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:29.816 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:29.816 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:29.816 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:29.818 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:29.818 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.813+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:29.821 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.817+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:29.821 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.817+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-06-15T09:39:29.821 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.817+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.817+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 freelist init 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 freelist _read_cfg 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2024-06-15T09:39:29.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 bluefs umount 2024-06-15T09:39:29.823 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:29.821+0000 7f68273deac0 1 bdev(0x55f5facec700 td/osd-mapper/1/block) close 2024-06-15T09:39:30.120 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.121+0000 7f68273deac0 1 bdev(0x55f5facec700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-06-15T09:39:30.120 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.121+0000 7f68273deac0 1 bdev(0x55f5facec700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:30.121 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.121+0000 7f68273deac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-06-15T09:39:30.121 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.121+0000 7f68273deac0 1 bluefs mount 2024-06-15T09:39:30.135 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.133+0000 7f68273deac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:30.145 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 1 bluefs mount shared_bdev_used = 9568256 2024-06-15T09:39:30.145 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.145+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.149+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:30.149 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.149+0000 7f68273deac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:30.480 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:30.481+0000 7f68273deac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:30.482 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-15T09:39:30.483 INFO:tasks.workunit.client.0.smithi116.stdout:00000000 0b 05 58 02 00 00 c8 8f 3c 4c c7 7a 42 91 a0 20 |..X.....= 1 )) 2024-06-15T09:39:31.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-15T09:39:31.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-15T09:39:31.903 INFO:tasks.workunit.client.0.smithi116.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:31.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-15T09:39:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.0BjpZZN0dO_the_val 2024-06-15T09:39:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:31.906 INFO:tasks.workunit.client.0.smithi116.stdout:Value dumped in: /tmp/tmp.0BjpZZN0dO_the_val 2024-06-15T09:39:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.0BjpZZN0dO_the_val' 2024-06-15T09:39:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.0BjpZZN0dO_the_val 2024-06-15T09:39:32.474 INFO:tasks.workunit.client.0.smithi116.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-15T09:39:32.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:32.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.0BjpZZN0dO_the_val 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-15T09:39:32.975 INFO:tasks.workunit.client.0.smithi116.stdout:0000073 2024-06-15T09:39:32.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-15T09:39:32.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:34.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.0BjpZZN0dO_the_val 2024-06-15T09:39:35.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:35.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:35.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-06-15T09:39:35.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-06-15T09:39:35.370 INFO:tasks.workunit.client.0.smithi116.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-06-15T09:39:35.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-06-15T09:39:35.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-06-15T09:39:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.397+0000 7f822b283ac0 1 bdev(0x55dc7f548000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bdev(0x55dc7f548000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-06-15T09:39:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bdev(0x55dc7f548700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:35.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bdev(0x55dc7f548700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:35.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-15T09:39:35.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.401+0000 7f822b283ac0 1 bluefs mount 2024-06-15T09:39:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.409+0000 7f822b283ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:35.420 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.417+0000 7f822b283ac0 1 bluefs mount shared_bdev_used = 0 2024-06-15T09:39:35.420 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:35.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.421+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 freelist init 2024-06-15T09:39:35.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 freelist _read_cfg 2024-06-15T09:39:35.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-06-15T09:39:35.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bluefs umount 2024-06-15T09:39:35.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.429+0000 7f822b283ac0 1 bdev(0x55dc7f548700 td/osd-mapper/2/block) close 2024-06-15T09:39:35.680 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.681+0000 7f822b283ac0 1 bdev(0x55dc7f548700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-06-15T09:39:35.680 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.681+0000 7f822b283ac0 1 bdev(0x55dc7f548700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-06-15T09:39:35.680 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.681+0000 7f822b283ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-06-15T09:39:35.681 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.681+0000 7f822b283ac0 1 bluefs mount 2024-06-15T09:39:35.685 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.685+0000 7f822b283ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-06-15T09:39:35.694 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.693+0000 7f822b283ac0 1 bluefs mount shared_bdev_used = 9568256 2024-06-15T09:39:35.694 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.693+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-06-15T09:39:35.697 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-06-15T09:39:35.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.697+0000 7f822b283ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-06-15T09:39:35.979 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:35.977+0000 7f822b283ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-06-15T09:39:35.982 INFO:tasks.workunit.client.0.smithi116.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-06-15T09:39:35.982 INFO:tasks.workunit.client.0.smithi116.stdout:00000000 0b 05 58 02 00 00 c8 8f 3c 4c c7 7a 42 91 a0 20 |..X.....= 1 )) 2024-06-15T09:39:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-06-15T09:39:37.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-06-15T09:39:37.503 INFO:tasks.workunit.client.0.smithi116.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:37.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-06-15T09:39:37.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.t475h02dYo_the_val 2024-06-15T09:39:37.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:37.507 INFO:tasks.workunit.client.0.smithi116.stdout:Value dumped in: /tmp/tmp.t475h02dYo_the_val 2024-06-15T09:39:37.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.t475h02dYo_the_val' 2024-06-15T09:39:37.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.t475h02dYo_the_val 2024-06-15T09:39:38.232 INFO:tasks.workunit.client.0.smithi116.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-06-15T09:39:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:38.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.t475h02dYo_the_val 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-06-15T09:39:38.583 INFO:tasks.workunit.client.0.smithi116.stdout:0000073 2024-06-15T09:39:38.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-15T09:39:38.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:39:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.t475h02dYo_the_val 2024-06-15T09:39:40.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:39:40.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-06-15T09:39:40.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-06-15T09:39:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:39:40.971 INFO:tasks.workunit.client.0.smithi116.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-06-15T09:39:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-06-15T09:39:40.972 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-06-15T09:39:40.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:40.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:39:40.977 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:39:40.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:39:40.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:39:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:39:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:39:40.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-06-15T09:39:40.980 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:39:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:39:41.023 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:41.021+0000 7f2bf60b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:41.026 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:41.025+0000 7f2bf60b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:41.027 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:41.025+0000 7f2bf60b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:39:41.235 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:39:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:39:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:41.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:41.570 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:41.569+0000 7f2bf60b77c0 -1 Falling back to public interface 2024-06-15T09:39:42.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:42.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:42.482 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:39:42.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:39:42.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:42.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:39:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:43.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:43.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:43.739 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:39:43.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:44.027 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:44.025+0000 7f2bf60b77c0 -1 osd.0 28 log_to_monitors true 2024-06-15T09:39:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:39:44.997 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:39:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:39:45.289 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6800/42168561,v1:127.0.0.1:6801/42168561] [v2:127.0.0.1:6802/42168561,v1:127.0.0.1:6803/42168561] exists,up 971fd802-e5b4-43cf-a728-c0cc4d756d19 2024-06-15T09:39:45.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:45.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:39:45.294 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:39:45.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:39:45.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:39:45.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:39:45.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-06-15T09:39:45.296 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:39:45.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:39:45.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:39:45.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-06-15T09:39:45.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:39:45.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:39:45.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:39:45.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:39:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:45.333+0000 7f45e0aa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:45.333+0000 7f45e0aa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:45.336 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:45.337+0000 7f45e0aa97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:45.549 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:39:45.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:39:45.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:45.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:39:45.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:45.882 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:45.881+0000 7f45e0aa97c0 -1 Falling back to public interface 2024-06-15T09:39:46.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:46.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:46.799 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:39:46.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:39:46.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:46.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:39:47.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:48.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:48.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:48.052 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:39:48.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:39:48.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:48.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:39:48.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:48.340 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:48.337+0000 7f45e0aa97c0 -1 osd.1 28 log_to_monitors true 2024-06-15T09:39:49.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:49.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:49.312 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:39:49.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:39:49.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:49.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:39:49.711 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/1583984031,v1:127.0.0.1:6811/1583984031] [v2:127.0.0.1:6812/1583984031,v1:127.0.0.1:6813/1583984031] exists,up 925c8271-eadd-4aae-8273-082413e222d2 2024-06-15T09:39:49.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:39:49.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:39:49.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:39:49.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:39:49.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:49.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:39:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-06-15T09:39:49.718 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:39:49.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c88f3c4c-c77a-4291-a020-c8c2bb3444ee --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20638/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:39:49.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:39:49.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:49.757+0000 7f4059b377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:49.773 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:49.773+0000 7f4059b377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:49.774 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:49.773+0000 7f4059b377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:39:49.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T09:39:49.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:39:49.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T09:39:49.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:39:49.966 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:39:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:39:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:39:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:39:50.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:50.310 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:50.309+0000 7f4059b377c0 -1 Falling back to public interface 2024-06-15T09:39:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:51.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:51.222 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:39:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:39:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:51.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:39:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:52.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:52.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:52.479 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:39:52.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:39:52.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:52.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:39:52.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:39:52.857 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:39:52.857+0000 7f4059b377c0 -1 osd.2 28 log_to_monitors true 2024-06-15T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:39:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:39:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:39:54.028 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 37 up_thru 37 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/868271439,v1:127.0.0.1:6819/868271439] [v2:127.0.0.1:6820/868271439,v1:127.0.0.1:6821/868271439] exists,up 45715acd-a9e2-43e8-bc50-20d68b10f58a 2024-06-15T09:39:54.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:39:54.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:39:54.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:39:54.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-06-15T09:39:55.031 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-06-15T09:39:55.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-06-15T09:39:55.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:55.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-06-15T09:39:55.176 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:55.176 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "19.0.0-2541-gc65be810", 2024-06-15T09:39:55.177 INFO:tasks.workunit.client.0.smithi116.stdout: "release": "squid", 2024-06-15T09:39:55.177 INFO:tasks.workunit.client.0.smithi116.stdout: "release_type": "rc" 2024-06-15T09:39:55.177 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-06-15T09:39:55.320 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:55.320 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "19.0.0-2541-gc65be810", 2024-06-15T09:39:55.320 INFO:tasks.workunit.client.0.smithi116.stdout: "release": "squid", 2024-06-15T09:39:55.320 INFO:tasks.workunit.client.0.smithi116.stdout: "release_type": "rc" 2024-06-15T09:39:55.320 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:55.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-06-15T09:39:55.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-06-15T09:39:55.461 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:55.461 INFO:tasks.workunit.client.0.smithi116.stdout: "version": "19.0.0-2541-gc65be810", 2024-06-15T09:39:55.461 INFO:tasks.workunit.client.0.smithi116.stdout: "release": "squid", 2024-06-15T09:39:55.461 INFO:tasks.workunit.client.0.smithi116.stdout: "release_type": "rc" 2024-06-15T09:39:55.461 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:55.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "objxxx", 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout: "clones": [ 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2024-06-15T09:39:55.503 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 1, 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap01" 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 2, 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap02" 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.504 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 3, 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 3, 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap13" 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.505 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 5, 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [ 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 4, 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap24" 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "id": 5, 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "name": "snap25" 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6, 2024-06-15T09:39:55.506 INFO:tasks.workunit.client.0.smithi116.stdout: "overlaps": [] 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: { 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "id": "head", 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "snapshots": [], 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: "size": 6 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: } 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout: ] 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:55.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-06-15T09:39:55.830 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is unset 2024-06-15T09:39:55.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-06-15T09:39:56.223 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is unset 2024-06-15T09:39:56.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-06-15T09:39:56.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-06-15T09:39:56.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-06-15T09:39:56.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-15T09:39:56.709 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:39:56.709 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 4 0 0 24 0 0 7 0 7 active+undersized+degraded 2024-06-15T09:39:49.608558+0000 26'7 36:51 [1,0] 1 [1,0] 1 26'7 2024-06-15T09:39:08.447085+0000 26'7 2024-06-15T09:39:08.447085+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:39:08.447085+0000 4 0 2024-06-15T09:39:56.709 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 activating 2024-06-15T09:39:54.227970+0000 0'0 39:32 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:39:56.709 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-06-15T09:39:49.433000+0000 0'0 35:13 [0,1] 0 [0,1] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:39:56.709 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-06-15T09:39:49.608364+0000 0'0 36:20 [1,0] 1 [1,0] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:39:56.710 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:39:56.710 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:39:56.710 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:39:56.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-06-15T09:39:58.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-15T09:39:58.863 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:39:58.863 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:39:58.863 INFO:tasks.workunit.client.0.smithi116.stdout: "must": true, 2024-06-15T09:39:58.863 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "0.000000" 2024-06-15T09:39:58.863 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:39:58.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-06-15T09:40:03.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-15T09:39:54.624676+0000 26'7 41:67 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-15T09:39:08.447085+0000 26'7 2024-06-15T09:39:08.447085+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:39:08.447085+0000 4 0 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.533019+0000 0'0 41:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.155389+0000 0'0 41:27 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.532511+0000 0'0 41:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:40:04.096 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:40:04.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:40:04.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-15T09:40:04.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:39:59.505+0000 7f45c3230640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-15T09:40:04.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-06-15T09:40:04.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-15T09:40:04.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-06-15T09:40:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-06-15T09:40:04.117 INFO:tasks.workunit.client.0.smithi116.stdout:prev count: 1 2024-06-15T09:40:04.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-06-15T09:40:04.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-06-15T09:40:04.246 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:40:04.247 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:40:04.247 INFO:tasks.workunit.client.0.smithi116.stdout: "must": true, 2024-06-15T09:40:04.247 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "0.000000" 2024-06-15T09:40:04.247 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:40:04.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-06-15T09:40:09.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-06-15T09:40:09.476 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:40:09.476 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-06-15T09:39:59.508973+0000 26'7 41:76 [1,2,0] 1 [1,2,0] 1 26'7 2024-06-15T09:39:59.508769+0000 26'7 2024-06-15T09:39:59.508769+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:39:59.508769+0000 4 0 2024-06-15T09:40:09.476 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.533019+0000 0'0 41:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:09.477 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.155389+0000 0'0 41:27 [2,0,1] 2 [2,0,1] 2 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:09.477 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:39:54.532511+0000 0'0 41:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-15T09:38:58.446981+0000 0'0 2024-06-15T09:38:58.446981+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:38:58.446981+0000 0 0 2024-06-15T09:40:09.477 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:40:09.477 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:40:09.477 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:40:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:40:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-15T09:40:09.493 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:39:59.505+0000 7f45c3230640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-06-15T09:40:09.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-06-15T09:40:09.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-06-15T09:40:09.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-06-15T09:40:09.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-06-15T09:40:09.497 INFO:tasks.workunit.client.0.smithi116.stdout:current count: 1 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:40:09.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:40:09.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:40:09.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-06-15T09:40:09.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-06-15T09:40:09.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-15T09:40:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-06-15T09:40:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-15T09:40:11.365 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-06-15T09:40:11.365 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-06-15T09:40:11.365 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-06-15T09:40:11.366 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-06-15T09:40:11.366 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-06-15T09:40:11.366 INFO:tasks.workunit.client.0.smithi116.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-06-15T09:40:11.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-06-15T09:40:11.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-06-15T09:40:11.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-06-15T09:40:11.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-15T09:40:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:40:12.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-15T09:40:12.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:40:12.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:40:12.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:40:12.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:40:12.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:40:12.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:40:12.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:40:12.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:40:12.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:40:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:40:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:40:12.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:40:12.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:40:12.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:40:12.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:40:12.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:40:12.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-15T09:40:12.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:40:12.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:12.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:40:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20638 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:40:12.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:40:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:40:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:40:12.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:40:12.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:40:12.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:40:12.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:40:12.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:40:12.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:40:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:40:12.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:40:12.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:40:12.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-15T09:40:12.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-06-15T09:40:12.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:40:12.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:12.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20638 2024-06-15T09:40:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20638 2024-06-15T09:40:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:40:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:40:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-15T09:40:12.589 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-15T09:40:12.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-15T09:40:12.643 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-06-15T09:40:12.644 DEBUG:teuthology.orchestra.run.smithi116:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-06-15T09:40:12.700 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-15T09:40:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-15T09:40:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:40:12.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-06-15T09:40:12.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-06-15T09:40:12.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-06-15T09:40:12.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none ' 2024-06-15T09:40:12.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-06-15T09:40:12.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-06-15T09:40:12.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-15T09:40:12.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-06-15T09:40:12.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_recovery_scrub_2' 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.37079 2024-06-15T09:40:12.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:40:12.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:40:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:40:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:40:12.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:40:12.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:40:12.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:40:12.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:40:12.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:40:12.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:40:12.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:40:12.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:40:12.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:40:12.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:40:12.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:40:12.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:40:12.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-15T09:40:12.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:40:12.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:12.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:12.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37079 2024-06-15T09:40:12.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:40:12.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:40:12.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-15T09:40:12.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:40:12.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:12.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:12.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37079 2024-06-15T09:40:12.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:40:12.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-15T09:40:12.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-15T09:40:12.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-15T09:40:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:13.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:13.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:13.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:40:13.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:40:13.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:40:13.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:40:13.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:40:13.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:40:13.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:40:13.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:40:13.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:40:13.322 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:40:13.322 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:40:13.323 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:13.323 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:13.323 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37079/ceph-mon.a.asok 2024-06-15T09:40:13.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:40:13.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37079/ceph-mon.a.asok config get fsid 2024-06-15T09:40:13.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:40:13.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:40:13.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:40:13.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:40:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37079/ceph-mon.a.asok 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:40:13.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37079/ceph-mon.a.asok config get mon_host 2024-06-15T09:40:13.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-06-15T09:40:13.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-15T09:40:13.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:40:13.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:40:13.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:40:13.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-15T09:40:13.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:40:14.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:40:14.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:14.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:14.411 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:14.411 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:14.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:14.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:14.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:40:14.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:40:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-15T09:40:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-15T09:40:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-15T09:40:14.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-15T09:40:14.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-06-15T09:40:14.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:40:14.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:14.455 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:40:14.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90963f82-6f1e-41c7-8e8c-d8f7cba4076e 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 90963f82-6f1e-41c7-8e8c-d8f7cba4076e' 2024-06-15T09:40:14.457 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 90963f82-6f1e-41c7-8e8c-d8f7cba4076e 2024-06-15T09:40:14.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:40:14.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+YW1mha1JHBAAQjhZXXSqntm33uXNVgHeRw== 2024-06-15T09:40:14.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+YW1mha1JHBAAQjhZXXSqntm33uXNVgHeRw=="}' 2024-06-15T09:40:14.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90963f82-6f1e-41c7-8e8c-d8f7cba4076e -i td/osd-recovery-scrub/0/new.json 2024-06-15T09:40:14.750 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:40:14.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-15T09:40:14.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB+YW1mha1JHBAAQjhZXXSqntm33uXNVgHeRw== --osd-uuid 90963f82-6f1e-41c7-8e8c-d8f7cba4076e 2024-06-15T09:40:14.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.801+0000 7f6fcd6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:14.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.817+0000 7f6fcd6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:14.818 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.817+0000 7f6fcd6357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.817+0000 7f6fcd6357c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:40:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.817+0000 7f6fcd6357c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:40:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.817+0000 7f6fcd6357c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:40:14.900 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:14.901+0000 7f6fcd6357c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-15T09:40:21.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-15T09:40:21.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:40:21.138 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:40:21.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:40:21.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:40:21.407 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:40:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:40:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:40:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-15T09:40:21.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:40:21.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:40:21.442 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:21.441+0000 7f22443de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:21.484 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:21.485+0000 7f22443de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:21.486 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:21.485+0000 7f22443de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:21.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:40:21.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:21.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:40:21.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:22.050 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:22.049+0000 7f22443de7c0 -1 Falling back to public interface 2024-06-15T09:40:22.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:22.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:22.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:40:22.900 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:40:22.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:22.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:40:23.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:24.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:24.139 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:40:24.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:40:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:40:24.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:24.739 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:24.737+0000 7f22443de7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:40:25.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:25.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:25.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:40:25.378 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:40:25.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:25.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:40:25.621 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/8603623,v1:127.0.0.1:6803/8603623] [v2:127.0.0.1:6804/8603623,v1:127.0.0.1:6805/8603623] exists,up 90963f82-6f1e-41c7-8e8c-d8f7cba4076e 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:40:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:25.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:25.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-15T09:40:25.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:40:25.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b8c76c7-071f-4d3d-9439-fd94d0cb7aec 2024-06-15T09:40:25.628 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 6b8c76c7-071f-4d3d-9439-fd94d0cb7aec 2024-06-15T09:40:25.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6b8c76c7-071f-4d3d-9439-fd94d0cb7aec' 2024-06-15T09:40:25.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:40:25.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJYW1m8LreJhAAVtR5rMownzr4TY/Q6mx4ug== 2024-06-15T09:40:25.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJYW1m8LreJhAAVtR5rMownzr4TY/Q6mx4ug=="}' 2024-06-15T09:40:25.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b8c76c7-071f-4d3d-9439-fd94d0cb7aec -i td/osd-recovery-scrub/1/new.json 2024-06-15T09:40:26.068 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:40:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-15T09:40:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCJYW1m8LreJhAAVtR5rMownzr4TY/Q6mx4ug== --osd-uuid 6b8c76c7-071f-4d3d-9439-fd94d0cb7aec 2024-06-15T09:40:26.126 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.125+0000 7f8b1bf497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:26.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.153+0000 7f8b1bf497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:26.154 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.153+0000 7f8b1bf497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:26.155 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.153+0000 7f8b1bf497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:40:26.155 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.153+0000 7f8b1bf497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:40:26.155 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.153+0000 7f8b1bf497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:40:26.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:26.229+0000 7f8b1bf497c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-15T09:40:32.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-15T09:40:32.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:40:32.897 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:40:32.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:40:32.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:40:33.168 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:40:33.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:40:33.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-15T09:40:33.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:40:33.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:40:33.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:40:33.203 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:33.200+0000 7fdf55d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:33.241 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:33.240+0000 7fdf55d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:33.242 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:33.240+0000 7fdf55d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:40:33.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:33.442 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:40:33.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:40:33.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:33.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:40:33.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:33.794 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:33.792+0000 7fdf55d417c0 -1 Falling back to public interface 2024-06-15T09:40:34.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:34.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:34.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:40:34.698 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:40:34.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:34.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:40:34.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:35.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:35.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:35.942 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:40:35.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:40:35.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:35.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:40:36.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:36.488 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:36.488+0000 7fdf55d417c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:40:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:37.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:37.183 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:40:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:40:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:37.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:40:37.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:38.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:38.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:38.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:40:38.424 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:40:38.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:38.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:40:38.661 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1771428299,v1:127.0.0.1:6811/1771428299] [v2:127.0.0.1:6812/1771428299,v1:127.0.0.1:6813/1771428299] exists,up 6b8c76c7-071f-4d3d-9439-fd94d0cb7aec 2024-06-15T09:40:38.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:40:38.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:40:38.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:38.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:40:38.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4 2024-06-15T09:40:38.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4' 2024-06-15T09:40:38.669 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4 2024-06-15T09:40:38.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:40:38.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWYW1m0hjqKBAA7oThXF8Tvq8i2DsqzOvMlA== 2024-06-15T09:40:38.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWYW1m0hjqKBAA7oThXF8Tvq8i2DsqzOvMlA=="}' 2024-06-15T09:40:38.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4 -i td/osd-recovery-scrub/2/new.json 2024-06-15T09:40:38.994 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:40:39.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-15T09:40:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCWYW1m0hjqKBAA7oThXF8Tvq8i2DsqzOvMlA== --osd-uuid 5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4 2024-06-15T09:40:39.049 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.048+0000 7f658fcc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:39.087 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.084+0000 7f658fcc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:39.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.088+0000 7f658fcc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:39.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.088+0000 7f658fcc97c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:40:39.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.088+0000 7f658fcc97c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:40:39.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.088+0000 7f658fcc97c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:40:39.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:39.152+0000 7f658fcc97c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-15T09:40:45.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-15T09:40:45.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:40:45.891 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:40:45.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T09:40:45.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:40:46.161 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:40:46.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T09:40:46.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-15T09:40:46.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:40:46.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:40:46.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:40:46.189 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:46.188+0000 7f8347a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:46.191 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:46.188+0000 7f8347a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:46.192 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:46.192+0000 7f8347a377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:40:46.416 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:40:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:40:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:40:46.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:46.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:46.744+0000 7f8347a377c0 -1 Falling back to public interface 2024-06-15T09:40:47.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:47.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:47.668 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:40:47.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:40:47.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:47.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:40:47.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:48.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:48.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:48.907 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:40:48.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:40:48.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:48.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:40:49.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:49.366 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:49.364+0000 7f8347a377c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:40:50.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:50.149 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:40:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:40:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:40:50.407 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2050319947,v1:127.0.0.1:6819/2050319947] [v2:127.0.0.1:6820/2050319947,v1:127.0.0.1:6821/2050319947] exists,up 5cfac5e2-2cc2-4da9-94a6-4b48b2075aa4 2024-06-15T09:40:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:40:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:40:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:40:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-15T09:40:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:40:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-06-15T09:40:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-15T09:40:50.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:40:50.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7d0e27e-a825-4c65-9ce5-21c99e269800 2024-06-15T09:40:50.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e7d0e27e-a825-4c65-9ce5-21c99e269800' 2024-06-15T09:40:50.414 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 e7d0e27e-a825-4c65-9ce5-21c99e269800 2024-06-15T09:40:50.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:40:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiYW1mgsC8GRAAZ+YG2SlO7LSliKWbeJx7kg== 2024-06-15T09:40:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiYW1mgsC8GRAAZ+YG2SlO7LSliKWbeJx7kg=="}' 2024-06-15T09:40:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7d0e27e-a825-4c65-9ce5-21c99e269800 -i td/osd-recovery-scrub/3/new.json 2024-06-15T09:40:50.729 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:40:50.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-15T09:40:50.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCiYW1mgsC8GRAAZ+YG2SlO7LSliKWbeJx7kg== --osd-uuid e7d0e27e-a825-4c65-9ce5-21c99e269800 2024-06-15T09:40:50.784 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.784+0000 7f7d985607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:50.831 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.828+0000 7f7d985607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:50.832 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.832+0000 7f7d985607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.832+0000 7f7d985607c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:40:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.832+0000 7f7d985607c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:40:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.832+0000 7f7d985607c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:40:50.913 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:50.912+0000 7f7d985607c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-15T09:40:57.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-15T09:40:57.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:40:57.814 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T09:40:57.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T09:40:57.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:40:58.084 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T09:40:58.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T09:40:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-06-15T09:40:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:40:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:40:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:40:58.117 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:58.116+0000 7f04917547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:58.119 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:58.120+0000 7f04917547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:58.121 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:58.120+0000 7f04917547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:40:58.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T09:40:58.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:40:58.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T09:40:58.355 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:40:58.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:40:58.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:40:58.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:58.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:40:58.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:58.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:40:58.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:40:58.697 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:40:58.696+0000 7f04917547c0 -1 Falling back to public interface 2024-06-15T09:40:59.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:40:59.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:40:59.616 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:40:59.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:40:59.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:40:59.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:40:59.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:41:00.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:41:00.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:41:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:41:00.849 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:41:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:41:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:41:01.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:41:01.358 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:41:01.356+0000 7f04917547c0 -1 osd.3 0 log_to_monitors true 2024-06-15T09:41:02.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:41:02.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:41:02.089 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:41:02.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:41:02.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:41:02.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:41:02.356 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1518700315,v1:127.0.0.1:6827/1518700315] [v2:127.0.0.1:6828/1518700315,v1:127.0.0.1:6829/1518700315] exists,up e7d0e27e-a825-4c65-9ce5-21c99e269800 2024-06-15T09:41:02.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:41:02.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:41:02.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:41:02.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-06-15T09:41:02.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-06-15T09:41:02.943 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' created 2024-06-15T09:41:02.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T09:41:03.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-06-15T09:41:03.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:41:03.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:41:03.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:41:03.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:41:03.969 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:41:03.969 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:41:03.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:41:03.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:41:03.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:41:04.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:41:04.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:41:04.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-06-15T09:41:04.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-06-15T09:41:04.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-06-15T09:41:04.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:41:04.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:41:04.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-15T09:41:04.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-15T09:41:04.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-06-15T09:41:04.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:41:04.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:41:04.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-06-15T09:41:04.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-06-15T09:41:04.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444' 2024-06-15T09:41:04.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:41:04.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T09:41:04.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-06-15T09:41:04.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-06-15T09:41:04.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444 3-85899345922' 2024-06-15T09:41:04.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:41:04.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-06-15T09:41:04.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:41:04.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:41:04.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-06-15T09:41:04.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:41:04.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-06-15T09:41:04.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-06-15T09:41:04.925 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836489 2024-06-15T09:41:04.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:41:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-06-15T09:41:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:41:05.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-15T09:41:05.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:41:05.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:41:05.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-15T09:41:05.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:41:05.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-15T09:41:05.147 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672967 2024-06-15T09:41:05.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-15T09:41:05.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:41:05.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-15T09:41:05.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:41:05.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:41:05.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-06-15T09:41:05.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:41:05.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-06-15T09:41:05.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:41:05.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-06-15T09:41:05.403 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509444 2024-06-15T09:41:05.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-06-15T09:41:05.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:41:05.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509444 2024-06-15T09:41:05.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:41:06.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:41:06.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:41:06.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-06-15T09:41:06.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:41:06.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:41:06.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-06-15T09:41:06.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T09:41:06.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-06-15T09:41:06.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:41:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-06-15T09:41:06.902 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345922 2024-06-15T09:41:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-06-15T09:41:06.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T09:41:07.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-06-15T09:41:07.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:41:07.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:41:07.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:41:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:41:07.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:41:07.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:41:07.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:41:07.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:41:07.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:41:07.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:41:07.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:41:07.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:41:07.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:41:07.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-06-15T09:41:07.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-06-15T09:41:07.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-06-15T09:41:08.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-06-15T09:41:08.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-15T09:41:08.422 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:41:08.423 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:41:04.084129+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-06-15T09:41:02.837369+0000 0'0 2024-06-15T09:41:02.837369+0000 0 0 periodic scrub scheduled @ 2024-06-16T12:40:39.434283+0000 0 0 2024-06-15T09:41:08.423 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:41:08.423 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:41:08.423 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:41:08.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.37079 bs=1M count=50 2024-06-15T09:41:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:50+0 records in 2024-06-15T09:41:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:50+0 records out 2024-06-15T09:41:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.169106 s, 310 MB/s 2024-06-15T09:41:08.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-06-15T09:41:08.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:08.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.37079 2024-06-15T09:41:10.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:10.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.37079 2024-06-15T09:41:12.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:12.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.37079 2024-06-15T09:41:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.37079 2024-06-15T09:41:15.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:15.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.37079 2024-06-15T09:41:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.37079 2024-06-15T09:41:19.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:19.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.37079 2024-06-15T09:41:21.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:21.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.37079 2024-06-15T09:41:22.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:22.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.37079 2024-06-15T09:41:24.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:24.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.37079 2024-06-15T09:41:26.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:26.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.37079 2024-06-15T09:41:27.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:27.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.37079 2024-06-15T09:41:29.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:29.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.37079 2024-06-15T09:41:31.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:31.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.37079 2024-06-15T09:41:32.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:32.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.37079 2024-06-15T09:41:34.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:34.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.37079 2024-06-15T09:41:36.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:36.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.37079 2024-06-15T09:41:38.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:38.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.37079 2024-06-15T09:41:39.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:39.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.37079 2024-06-15T09:41:41.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:41.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.37079 2024-06-15T09:41:43.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:43.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.37079 2024-06-15T09:41:45.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:45.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.37079 2024-06-15T09:41:47.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:47.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.37079 2024-06-15T09:41:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.37079 2024-06-15T09:41:50.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:50.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.37079 2024-06-15T09:41:52.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:52.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.37079 2024-06-15T09:41:54.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:54.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.37079 2024-06-15T09:41:55.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:55.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.37079 2024-06-15T09:41:57.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:57.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.37079 2024-06-15T09:41:59.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:41:59.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.37079 2024-06-15T09:42:00.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:00.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.37079 2024-06-15T09:42:02.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:02.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.37079 2024-06-15T09:42:04.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:04.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.37079 2024-06-15T09:42:06.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:06.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.37079 2024-06-15T09:42:07.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:07.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.37079 2024-06-15T09:42:09.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:09.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.37079 2024-06-15T09:42:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.37079 2024-06-15T09:42:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:12.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.37079 2024-06-15T09:42:14.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:14.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.37079 2024-06-15T09:42:16.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:16.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.37079 2024-06-15T09:42:17.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:17.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.37079 2024-06-15T09:42:19.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:19.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.37079 2024-06-15T09:42:21.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:21.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.37079 2024-06-15T09:42:23.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:23.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.37079 2024-06-15T09:42:24.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:24.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.37079 2024-06-15T09:42:26.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:26.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.37079 2024-06-15T09:42:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.37079 2024-06-15T09:42:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.37079 2024-06-15T09:42:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:31.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.37079 2024-06-15T09:42:33.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:33.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.37079 2024-06-15T09:42:34.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:34.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.37079 2024-06-15T09:42:36.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:36.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.37079 2024-06-15T09:42:38.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:38.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.37079 2024-06-15T09:42:40.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:40.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.37079 2024-06-15T09:42:41.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:41.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.37079 2024-06-15T09:42:43.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:43.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.37079 2024-06-15T09:42:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.37079 2024-06-15T09:42:46.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:46.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.37079 2024-06-15T09:42:48.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:48.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.37079 2024-06-15T09:42:50.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:50.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.37079 2024-06-15T09:42:52.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:52.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.37079 2024-06-15T09:42:53.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:53.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.37079 2024-06-15T09:42:55.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:55.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.37079 2024-06-15T09:42:57.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:57.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.37079 2024-06-15T09:42:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:42:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.37079 2024-06-15T09:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.37079 2024-06-15T09:43:02.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:02.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.37079 2024-06-15T09:43:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:04.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.37079 2024-06-15T09:43:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.37079 2024-06-15T09:43:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:07.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.37079 2024-06-15T09:43:09.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:09.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.37079 2024-06-15T09:43:11.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:11.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.37079 2024-06-15T09:43:13.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:13.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.37079 2024-06-15T09:43:15.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:15.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.37079 2024-06-15T09:43:16.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:16.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.37079 2024-06-15T09:43:18.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:18.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.37079 2024-06-15T09:43:20.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:20.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.37079 2024-06-15T09:43:22.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:22.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.37079 2024-06-15T09:43:23.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:23.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.37079 2024-06-15T09:43:25.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:25.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.37079 2024-06-15T09:43:27.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:27.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.37079 2024-06-15T09:43:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:28.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.37079 2024-06-15T09:43:30.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:30.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.37079 2024-06-15T09:43:32.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:32.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.37079 2024-06-15T09:43:33.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:33.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.37079 2024-06-15T09:43:35.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:35.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.37079 2024-06-15T09:43:37.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:37.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.37079 2024-06-15T09:43:39.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:39.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.37079 2024-06-15T09:43:40.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:40.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.37079 2024-06-15T09:43:42.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:42.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.37079 2024-06-15T09:43:44.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:44.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.37079 2024-06-15T09:43:46.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:46.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.37079 2024-06-15T09:43:47.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:47.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.37079 2024-06-15T09:43:49.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:49.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.37079 2024-06-15T09:43:51.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:51.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.37079 2024-06-15T09:43:53.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:53.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.37079 2024-06-15T09:43:54.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:54.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.37079 2024-06-15T09:43:56.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:56.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.37079 2024-06-15T09:43:58.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:58.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.37079 2024-06-15T09:43:59.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-06-15T09:43:59.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.37079 2024-06-15T09:44:01.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.37079 2024-06-15T09:44:01.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-06-15T09:44:01.907 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 4 2024-06-15T09:44:01.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-06-15T09:44:01.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-06-15T09:44:01.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-15T09:44:01.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-15T09:44:01.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-15T09:44:01.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-15T09:44:02.190 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:44:02.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-15T09:44:04.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-06-15T09:44:04.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-06-15T09:44:04.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-06-15T09:44:04.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-15T09:44:04.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-15T09:44:04.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-15T09:44:04.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-15T09:44:04.440 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:44:04.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-15T09:44:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-06-15T09:44:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-06-15T09:44:06.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-06-15T09:44:06.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-15T09:44:06.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-15T09:44:06.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-15T09:44:06.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-15T09:44:06.687 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:44:06.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-06-15T09:44:08.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-06-15T09:44:08.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-06-15T09:44:08.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-06-15T09:44:08.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-06-15T09:44:08.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-06-15T09:44:08.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-06-15T09:44:08.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-06-15T09:44:08.989 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:44:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-06-15T09:44:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-06-15T09:44:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-06-15T09:44:09.270 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:44:09.271 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-06-15T09:44:05.159841+0000 24'1300 27:1323 [1,0,2,3] 1 [1,0] 1 0'0 2024-06-15T09:41:02.837369+0000 0'0 2024-06-15T09:41:02.837369+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:44:09.271 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:44:09.271 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:44:09.271 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:44:09.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-06-15T09:44:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-06-15T09:44:19.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:44:19.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:44:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:44:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:44:19.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:44:24.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:44:24.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-06-15T09:44:24.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-06-15T09:44:24.610 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-06-15T09:44:24.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-06-15T09:44:24.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:05.998+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:06.966+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:07.974+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:12.058+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:14.006+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.615 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:14.962+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:18.914+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:19.938+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:20.958+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:21.970+0000 7f223d392640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-15T09:44:24.630 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:05.938+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:11.890+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:13.854+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:18.818+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:22.898+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:23.870+0000 7fdf4ecf5640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-15T09:44:24.634 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:11.430+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.634 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:12.454+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.634 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:13.426+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.634 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:14.422+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:15.442+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:17.422+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:21.458+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:23.382+0000 7f83409eb640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-15T09:44:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:06.458+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:08.470+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:11.366+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:14.338+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.637 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:21.450+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.638 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:22.410+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.638 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:44:23.378+0000 7f048a708640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-06-15T09:44:24.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-06-15T09:44:24.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-06-15T09:44:24.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-06-15T09:44:24.639 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-06-15T09:44:24.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-06-15T09:44:24.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-06-15T09:44:24.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-15T09:44:24.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-06-15T09:44:24.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-06-15T09:44:24.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-06-15T09:44:24.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-15T09:44:24.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-06-15T09:44:24.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-06-15T09:44:24.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-06-15T09:44:24.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-15T09:44:24.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-06-15T09:44:24.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-06-15T09:44:24.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:24.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-06-15T09:44:24.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-06-15T09:44:24.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-06-15T09:44:24.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-06-15T09:44:24.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:44:24.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:44:24.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:44:24.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:44:24.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:44:24.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:44:24.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:44:24.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:44:24.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:44:24.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:44:24.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:44:24.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:44:24.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:44:24.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:44:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:44:24.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:44:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:44:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:44:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-15T09:44:24.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:44:24.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:24.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:24.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37079 2024-06-15T09:44:24.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:44:24.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:44:24.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-06-15T09:44:24.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-06-15T09:44:24.710 INFO:tasks.workunit.client.0.smithi116.stdout:TEST PASSED 2024-06-15T09:44:24.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.37079 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-06-15T09:44:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:44:24.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:44:24.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:44:24.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:44:24.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:44:24.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-15T09:44:24.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:44:24.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:24.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37079 2024-06-15T09:44:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:44:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:44:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-06-15T09:44:24.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:44:24.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:24.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:24.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.37079 2024-06-15T09:44:24.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-06-15T09:44:24.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:44:25.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:44:25.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:44:25.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:44:25.120 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:44:25.121 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:25.121 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:25.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:25.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-06-15T09:44:25.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:44:25.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:44:25.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:44:25.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:44:25.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:44:25.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:44:25.349 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:44:25.349 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:44:25.349 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37079/ceph-mon.a.asok 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:44:25.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37079/ceph-mon.a.asok config get fsid 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:44:25.467 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:44:25.468 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:25.468 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:25.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.37079/ceph-mon.a.asok 2024-06-15T09:44:25.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:44:25.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.37079/ceph-mon.a.asok config get mon_host 2024-06-15T09:44:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-06-15T09:44:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-06-15T09:44:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:44:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:44:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:44:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-06-15T09:44:25.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:44:26.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:44:26.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:44:26.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:44:26.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:44:26.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:26.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:26.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:26.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:44:26.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:44:26.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-06-15T09:44:26.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-06-15T09:44:26.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-06-15T09:44:26.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-06-15T09:44:26.484 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-06-15T09:44:26.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-06-15T09:44:26.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:26.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:44:26.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:44:26.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf23006c-aae5-498b-bc12-715d2c635b3d 2024-06-15T09:44:26.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bf23006c-aae5-498b-bc12-715d2c635b3d' 2024-06-15T09:44:26.493 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 bf23006c-aae5-498b-bc12-715d2c635b3d 2024-06-15T09:44:26.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:44:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6Ym1mdkqCHhAAShDLDiUIHy/XEAMEeFVGSw== 2024-06-15T09:44:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6Ym1mdkqCHhAAShDLDiUIHy/XEAMEeFVGSw=="}' 2024-06-15T09:44:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf23006c-aae5-498b-bc12-715d2c635b3d -i td/osd-recovery-scrub/0/new.json 2024-06-15T09:44:26.742 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:44:26.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-06-15T09:44:26.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB6Ym1mdkqCHhAAShDLDiUIHy/XEAMEeFVGSw== --osd-uuid bf23006c-aae5-498b-bc12-715d2c635b3d 2024-06-15T09:44:26.798 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.798+0000 7f3e07dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:26.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.802+0000 7f3e07dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:26.801 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.802+0000 7f3e07dbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:26.801 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.802+0000 7f3e07dbd7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:44:26.801 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.802+0000 7f3e07dbd7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:44:26.802 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.802+0000 7f3e07dbd7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-06-15T09:44:26.893 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:26.894+0000 7f3e07dbd7c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-06-15T09:44:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-06-15T09:44:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:44:32.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:44:32.892 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:44:32.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:44:33.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:44:33.164 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:44:33.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:44:33.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:33.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:44:33.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:44:33.201 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:33.202+0000 7fa9abf6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:33.232 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:33.234+0000 7fa9abf6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:33.234 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:33.238+0000 7fa9abf6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:33.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:44:33.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:44:33.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:44:33.424 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:33.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:44:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:33.797 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:33.798+0000 7fa9abf6c7c0 -1 Falling back to public interface 2024-06-15T09:44:34.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:34.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:34.669 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:44:34.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:44:34.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:34.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:44:34.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:35.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:35.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:35.911 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:44:35.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:44:35.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:35.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:44:36.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:36.480 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:36.482+0000 7fa9abf6c7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:44:37.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:37.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:37.152 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:44:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:44:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:44:37.390 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/947846815,v1:127.0.0.1:6803/947846815] [v2:127.0.0.1:6804/947846815,v1:127.0.0.1:6805/947846815] exists,up bf23006c-aae5-498b-bc12-715d2c635b3d 2024-06-15T09:44:37.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:44:37.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:44:37.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-06-15T09:44:37.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:44:37.395 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:44:37.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-06-15T09:44:37.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:44:37.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be725d4b-61b6-4ee9-8efd-05811ebb71ee 2024-06-15T09:44:37.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 be725d4b-61b6-4ee9-8efd-05811ebb71ee' 2024-06-15T09:44:37.398 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 be725d4b-61b6-4ee9-8efd-05811ebb71ee 2024-06-15T09:44:37.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:44:37.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFYm1msg7cGBAAcse3HIWlCgIpq/2ft+iaRg== 2024-06-15T09:44:37.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFYm1msg7cGBAAcse3HIWlCgIpq/2ft+iaRg=="}' 2024-06-15T09:44:37.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be725d4b-61b6-4ee9-8efd-05811ebb71ee -i td/osd-recovery-scrub/1/new.json 2024-06-15T09:44:37.758 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:44:37.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-06-15T09:44:37.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCFYm1msg7cGBAAcse3HIWlCgIpq/2ft+iaRg== --osd-uuid be725d4b-61b6-4ee9-8efd-05811ebb71ee 2024-06-15T09:44:37.817 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.818+0000 7f37eef8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:37.851 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.854+0000 7f37eef8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:37.853 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.854+0000 7f37eef8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:37.853 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.854+0000 7f37eef8c7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:44:37.853 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.854+0000 7f37eef8c7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:44:37.853 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.854+0000 7f37eef8c7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-06-15T09:44:37.924 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:37.926+0000 7f37eef8c7c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-06-15T09:44:44.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-06-15T09:44:44.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:44:44.444 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:44:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:44:44.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:44:44.715 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:44:44.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:44:44.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:44.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:44:44.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:44:44.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:44:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:44.754+0000 7f2c442827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:44.780 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:44.782+0000 7f2c442827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:44.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:44.782+0000 7f2c442827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:44:44.981 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:44:44.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:44.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:44:45.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:45.333 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:45.334+0000 7f2c442827c0 -1 Falling back to public interface 2024-06-15T09:44:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:46.255 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:44:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:44:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:44:46.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:47.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:47.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:47.496 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:44:47.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:44:47.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:47.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:44:47.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:48.085 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:48.086+0000 7f2c442827c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:44:48.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:48.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:48.856 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:44:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:44:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:44:49.119 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3829995779,v1:127.0.0.1:6811/3829995779] [v2:127.0.0.1:6812/3829995779,v1:127.0.0.1:6813/3829995779] exists,up be725d4b-61b6-4ee9-8efd-05811ebb71ee 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T09:44:49.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:44:49.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:44:49.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:44:49.125 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 68c77f7b-d30c-4ecc-a60c-a39f844ee0d5 2024-06-15T09:44:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=68c77f7b-d30c-4ecc-a60c-a39f844ee0d5 2024-06-15T09:44:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 68c77f7b-d30c-4ecc-a60c-a39f844ee0d5' 2024-06-15T09:44:49.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:44:49.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRYm1meqqtCBAAiI6ohH50AjXwFuz+mHfWUg== 2024-06-15T09:44:49.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRYm1meqqtCBAAiI6ohH50AjXwFuz+mHfWUg=="}' 2024-06-15T09:44:49.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 68c77f7b-d30c-4ecc-a60c-a39f844ee0d5 -i td/osd-recovery-scrub/2/new.json 2024-06-15T09:44:49.466 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:44:49.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-06-15T09:44:49.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCRYm1meqqtCBAAiI6ohH50AjXwFuz+mHfWUg== --osd-uuid 68c77f7b-d30c-4ecc-a60c-a39f844ee0d5 2024-06-15T09:44:49.523 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.526+0000 7f307e8e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:49.560 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.562+0000 7f307e8e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:49.561 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.562+0000 7f307e8e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:49.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.562+0000 7f307e8e07c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:44:49.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.562+0000 7f307e8e07c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:44:49.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.562+0000 7f307e8e07c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-06-15T09:44:49.637 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:49.638+0000 7f307e8e07c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-06-15T09:44:56.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-06-15T09:44:56.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:44:56.047 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:44:56.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T09:44:56.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:44:56.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T09:44:56.315 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:44:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:44:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:44:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:44:56.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:44:56.350 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:56.354+0000 7f68ca5357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:56.382 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:56.382+0000 7f68ca5357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:56.386+0000 7f68ca5357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:44:56.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T09:44:56.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:44:56.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:44:56.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:56.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:44:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:56.940 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:56.942+0000 7f68ca5357c0 -1 Falling back to public interface 2024-06-15T09:44:57.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:44:57.864 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:44:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:44:58.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:59.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:44:59.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:44:59.104 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:44:59.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:44:59.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:44:59.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:44:59.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:44:59.527 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:44:59.530+0000 7f68ca5357c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:45:00.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:00.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:00.337 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:45:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:00.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:45:00.590 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1894244073,v1:127.0.0.1:6819/1894244073] [v2:127.0.0.1:6820/1894244073,v1:127.0.0.1:6821/1894244073] exists,up 68c77f7b-d30c-4ecc-a60c-a39f844ee0d5 2024-06-15T09:45:00.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:45:00.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:45:00.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:45:00.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:45:00.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:00.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:45:00.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:45:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:45:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:45:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-06-15T09:45:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-06-15T09:45:00.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:45:00.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:45:00.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:45:00.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:45:00.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:45:00.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:45:00.602 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:45:00.602 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:45:00.602 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:45:00.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:00.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:00.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:45:00.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:45:00.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:45:00.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:45:00.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:45:00.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:45:00.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:45:00.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:45:00.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:45:00.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:45:00.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:45:00.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-06-15T09:45:00.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:45:00.606 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 9dc38465-000a-4b78-96db-0f6c36aaf934 2024-06-15T09:45:00.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9dc38465-000a-4b78-96db-0f6c36aaf934 2024-06-15T09:45:00.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9dc38465-000a-4b78-96db-0f6c36aaf934' 2024-06-15T09:45:00.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:45:00.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcYm1mIcH3JBAA2mzkNvyh23IEr9v88KekNw== 2024-06-15T09:45:00.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcYm1mIcH3JBAA2mzkNvyh23IEr9v88KekNw=="}' 2024-06-15T09:45:00.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9dc38465-000a-4b78-96db-0f6c36aaf934 -i td/osd-recovery-scrub/3/new.json 2024-06-15T09:45:00.943 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:00.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-06-15T09:45:00.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCcYm1mIcH3JBAA2mzkNvyh23IEr9v88KekNw== --osd-uuid 9dc38465-000a-4b78-96db-0f6c36aaf934 2024-06-15T09:45:00.998 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:00.998+0000 7f42b33827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:01.054 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.054+0000 7f42b33827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:01.058 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.058+0000 7f42b33827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:01.058 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.058+0000 7f42b33827c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:45:01.058 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.058+0000 7f42b33827c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:45:01.058 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.058+0000 7f42b33827c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-06-15T09:45:01.146 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:01.150+0000 7f42b33827c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-06-15T09:45:07.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-06-15T09:45:07.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:45:07.812 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T09:45:07.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T09:45:07.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:45:08.087 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T09:45:08.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T09:45:08.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:08.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:45:08.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:45:08.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:45:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:08.125+0000 7fc136d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:08.152 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:08.153+0000 7fc136d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:08.154 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:08.153+0000 7fc136d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:45:08.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:08.348 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:45:08.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:45:08.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:08.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:45:08.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:08.713 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:08.713+0000 7fc136d577c0 -1 Falling back to public interface 2024-06-15T09:45:09.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:09.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:09.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:45:09.623 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:45:09.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:09.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:45:09.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:10.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:10.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:10.863 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:45:10.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:45:10.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:10.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:45:11.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:11.433 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:11.433+0000 7fc136d577c0 -1 osd.3 0 log_to_monitors true 2024-06-15T09:45:12.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:12.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:12.095 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:12.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:45:12.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:12.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T09:45:12.397 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3413873299,v1:127.0.0.1:6827/3413873299] [v2:127.0.0.1:6828/3413873299,v1:127.0.0.1:6829/3413873299] exists,up 9dc38465-000a-4b78-96db-0f6c36aaf934 2024-06-15T09:45:12.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:45:12.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:45:12.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:45:12.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:45:12.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:45:12.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:45:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:45:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-06-15T09:45:12.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:45:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c67ae299-fc52-40f9-be80-bbe27b7bcd3b 2024-06-15T09:45:12.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c67ae299-fc52-40f9-be80-bbe27b7bcd3b' 2024-06-15T09:45:12.405 INFO:tasks.workunit.client.0.smithi116.stdout:add osd4 c67ae299-fc52-40f9-be80-bbe27b7bcd3b 2024-06-15T09:45:12.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:45:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoYm1mwIBwGRAA+EXc4+kmx+eoUwpktBJeCw== 2024-06-15T09:45:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoYm1mwIBwGRAA+EXc4+kmx+eoUwpktBJeCw=="}' 2024-06-15T09:45:12.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c67ae299-fc52-40f9-be80-bbe27b7bcd3b -i td/osd-recovery-scrub/4/new.json 2024-06-15T09:45:12.805 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:45:12.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-06-15T09:45:12.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCoYm1mwIBwGRAA+EXc4+kmx+eoUwpktBJeCw== --osd-uuid c67ae299-fc52-40f9-be80-bbe27b7bcd3b 2024-06-15T09:45:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.865+0000 7fcb4144a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:12.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.897+0000 7fcb4144a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:12.899 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.901+0000 7fcb4144a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:12.900 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.901+0000 7fcb4144a7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-15T09:45:12.900 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.901+0000 7fcb4144a7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-15T09:45:12.900 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.901+0000 7fcb4144a7c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-06-15T09:45:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:12.985+0000 7fcb4144a7c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-06-15T09:45:19.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-06-15T09:45:19.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:45:19.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-15T09:45:19.868 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd4 key to auth repository 2024-06-15T09:45:19.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:45:20.134 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.4 2024-06-15T09:45:20.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-15T09:45:20.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:20.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:45:20.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:45:20.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:45:20.171 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:20.173+0000 7ff1dad807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:20.199 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:20.201+0000 7ff1dad807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:20.201 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:20.201+0000 7ff1dad807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:20.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:45:20.399 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:45:20.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:20.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T09:45:20.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:20.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:20.757+0000 7ff1dad807c0 -1 Falling back to public interface 2024-06-15T09:45:21.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:21.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:21.655 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:45:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:45:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T09:45:21.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:22.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:22.896 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:45:22.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:22.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:45:22.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:22.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T09:45:23.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:23.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:23.421+0000 7ff1dad807c0 -1 osd.4 0 log_to_monitors true 2024-06-15T09:45:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:24.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:45:24.141 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:24.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:24.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T09:45:24.386 INFO:tasks.workunit.client.0.smithi116.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/517036193,v1:127.0.0.1:6835/517036193] [v2:127.0.0.1:6836/517036193,v1:127.0.0.1:6837/517036193] exists,up c67ae299-fc52-40f9-be80-bbe27b7bcd3b 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-15T09:45:24.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:45:24.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:45:24.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-06-15T09:45:24.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:45:24.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f514cf55-bca9-4e97-92ad-9f1fe0a2ee77 2024-06-15T09:45:24.394 INFO:tasks.workunit.client.0.smithi116.stdout:add osd5 f514cf55-bca9-4e97-92ad-9f1fe0a2ee77 2024-06-15T09:45:24.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f514cf55-bca9-4e97-92ad-9f1fe0a2ee77' 2024-06-15T09:45:24.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:45:24.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0Ym1m12nZGBAAVYkXr8gITIXaaEnEpJ+lFg== 2024-06-15T09:45:24.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0Ym1m12nZGBAAVYkXr8gITIXaaEnEpJ+lFg=="}' 2024-06-15T09:45:24.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f514cf55-bca9-4e97-92ad-9f1fe0a2ee77 -i td/osd-recovery-scrub/5/new.json 2024-06-15T09:45:24.778 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:45:24.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-06-15T09:45:24.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC0Ym1m12nZGBAAVYkXr8gITIXaaEnEpJ+lFg== --osd-uuid f514cf55-bca9-4e97-92ad-9f1fe0a2ee77 2024-06-15T09:45:24.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.837+0000 7f514adfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:24.871 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.873+0000 7f514adfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:24.873 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.873+0000 7f514adfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:24.873 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.873+0000 7f514adfd7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-15T09:45:24.873 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.873+0000 7f514adfd7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-15T09:45:24.874 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.873+0000 7f514adfd7c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-06-15T09:45:24.987 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:24.989+0000 7f514adfd7c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-06-15T09:45:32.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-06-15T09:45:32.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:45:32.205 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd5 key to auth repository 2024-06-15T09:45:32.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-15T09:45:32.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:45:32.478 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.5 2024-06-15T09:45:32.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-15T09:45:32.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:45:32.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:45:32.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:32.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:45:32.517 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:32.517+0000 7f2296faa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:32.549+0000 7f2296faa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:32.549+0000 7f2296faa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:32.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-15T09:45:32.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:45:32.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T09:45:32.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:45:32.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:45:32.751 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:45:32.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:32.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:45:32.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:32.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T09:45:33.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:33.105 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:33.105+0000 7f2296faa7c0 -1 Falling back to public interface 2024-06-15T09:45:34.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:34.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:34.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:45:34.015 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:45:34.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:34.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T09:45:34.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:35.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:35.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:35.255 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:45:35.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:45:35.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:35.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T09:45:35.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:35.857 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:35.857+0000 7f2296faa7c0 -1 osd.5 0 log_to_monitors true 2024-06-15T09:45:36.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:36.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:36.497 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:36.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:45:36.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:36.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T09:45:36.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:37.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:37.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:37.741 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:45:37.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:45:37.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:37.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T09:45:37.998 INFO:tasks.workunit.client.0.smithi116.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2187630304,v1:127.0.0.1:6843/2187630304] [v2:127.0.0.1:6844/2187630304,v1:127.0.0.1:6845/2187630304] exists,up f514cf55-bca9-4e97-92ad-9f1fe0a2ee77 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:45:38.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:45:38.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:45:38.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:45:38.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:45:38.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:45:38.003 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:45:38.003 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:45:38.003 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:45:38.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:45:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-06-15T09:45:38.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:45:38.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04ff40a1-37f6-4a87-88f0-1e90a4c494fb 2024-06-15T09:45:38.005 INFO:tasks.workunit.client.0.smithi116.stdout:add osd6 04ff40a1-37f6-4a87-88f0-1e90a4c494fb 2024-06-15T09:45:38.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 04ff40a1-37f6-4a87-88f0-1e90a4c494fb' 2024-06-15T09:45:38.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:45:38.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCYm1my0efARAAerbtClrbTFXmtwzQL/ynfg== 2024-06-15T09:45:38.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCYm1my0efARAAerbtClrbTFXmtwzQL/ynfg=="}' 2024-06-15T09:45:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04ff40a1-37f6-4a87-88f0-1e90a4c494fb -i td/osd-recovery-scrub/6/new.json 2024-06-15T09:45:38.331 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:45:38.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-06-15T09:45:38.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDCYm1my0efARAAerbtClrbTFXmtwzQL/ynfg== --osd-uuid 04ff40a1-37f6-4a87-88f0-1e90a4c494fb 2024-06-15T09:45:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.385+0000 7f49f1a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:38.398 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.401+0000 7f49f1a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.401+0000 7f49f1a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.401+0000 7f49f1a357c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-15T09:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.401+0000 7f49f1a357c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-15T09:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.401+0000 7f49f1a357c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-06-15T09:45:38.541 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:38.541+0000 7f49f1a357c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-06-15T09:45:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-06-15T09:45:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:45:45.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-15T09:45:45.180 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd6 key to auth repository 2024-06-15T09:45:45.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:45:45.454 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.6 2024-06-15T09:45:45.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-15T09:45:45.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:45:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:45:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:45:45.491 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:45.493+0000 7f74499b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:45.525 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:45.525+0000 7f74499b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:45.527 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:45.529+0000 7f74499b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:45.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:45:45.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:45.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:45:45.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:45.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T09:45:45.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:46.085 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:46.085+0000 7f74499b97c0 -1 Falling back to public interface 2024-06-15T09:45:46.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:46.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:46.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:45:46.993 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:45:46.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:46.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T09:45:47.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:48.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:48.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:48.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:45:48.235 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:45:48.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:48.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T09:45:48.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:48.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:48.801+0000 7f74499b97c0 -1 osd.6 0 log_to_monitors true 2024-06-15T09:45:49.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:49.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:49.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:45:49.476 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:45:49.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:49.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T09:45:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:50.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:45:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:50.719 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:45:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:45:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:50.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T09:45:51.001 INFO:tasks.workunit.client.0.smithi116.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3178437753,v1:127.0.0.1:6851/3178437753] [v2:127.0.0.1:6852/3178437753,v1:127.0.0.1:6853/3178437753] exists,up 04ff40a1-37f6-4a87-88f0-1e90a4c494fb 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-06-15T09:45:51.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:45:51.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-06-15T09:45:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-06-15T09:45:51.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:45:51.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=967efcef-3665-4259-aa41-0b3fe81ebf98 2024-06-15T09:45:51.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 967efcef-3665-4259-aa41-0b3fe81ebf98' 2024-06-15T09:45:51.008 INFO:tasks.workunit.client.0.smithi116.stdout:add osd7 967efcef-3665-4259-aa41-0b3fe81ebf98 2024-06-15T09:45:51.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:45:51.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPYm1mC5zNARAApAZ5qW7BTdg44M/2p7ZevQ== 2024-06-15T09:45:51.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPYm1mC5zNARAApAZ5qW7BTdg44M/2p7ZevQ=="}' 2024-06-15T09:45:51.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 967efcef-3665-4259-aa41-0b3fe81ebf98 -i td/osd-recovery-scrub/7/new.json 2024-06-15T09:45:51.358 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:45:51.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-06-15T09:45:51.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDPYm1mC5zNARAApAZ5qW7BTdg44M/2p7ZevQ== --osd-uuid 967efcef-3665-4259-aa41-0b3fe81ebf98 2024-06-15T09:45:51.417 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.417+0000 7fe4c35577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.453+0000 7fe4c35577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:51.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.453+0000 7fe4c35577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:51.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.457+0000 7fe4c35577c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-15T09:45:51.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.457+0000 7fe4c35577c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-15T09:45:51.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.457+0000 7fe4c35577c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-06-15T09:45:51.551 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:51.553+0000 7fe4c35577c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-06-15T09:45:58.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-06-15T09:45:58.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:45:58.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-15T09:45:58.492 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd7 key to auth repository 2024-06-15T09:45:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:45:58.761 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.7 2024-06-15T09:45:58.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-15T09:45:58.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:45:58.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=6eb1f828-c63c-42a9-a47f-7c3d0a78afef --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37079/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-06-15T09:45:58.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:45:58.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:45:58.797 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:58.797+0000 7ff0ac73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:58.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:58.801+0000 7ff0ac73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:58.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:58.801+0000 7ff0ac73b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:45:58.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-15T09:45:58.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:45:58.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-15T09:45:58.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:45:58.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:45:58.990 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:45:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:45:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:45:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:45:58.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T09:45:59.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:45:59.349 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:45:59.349+0000 7ff0ac73b7c0 -1 Falling back to public interface 2024-06-15T09:46:00.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:46:00.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:46:00.224 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:46:00.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:46:00.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:46:00.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T09:46:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:46:01.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:46:01.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:46:01.460 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:46:01.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:46:01.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:46:01.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T09:46:01.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:46:02.103 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:46:02.105+0000 7ff0ac73b7c0 -1 osd.7 0 log_to_monitors true 2024-06-15T09:46:02.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:46:02.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:46:02.804 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:46:02.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:46:02.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:46:02.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T09:46:03.162 INFO:tasks.workunit.client.0.smithi116.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2064891836,v1:127.0.0.1:6859/2064891836] [v2:127.0.0.1:6860/2064891836,v1:127.0.0.1:6861/2064891836] exists,up 967efcef-3665-4259-aa41-0b3fe81ebf98 2024-06-15T09:46:03.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:46:03.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:46:03.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:46:03.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-06-15T09:46:03.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-06-15T09:46:03.516 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'test' created 2024-06-15T09:46:03.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T09:46:04.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-06-15T09:46:04.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:46:04.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:46:04.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:46:04.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:46:04.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:46:04.538 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:46:04.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:46:04.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:46:04.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:46:04.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T09:46:05.203 INFO:tasks.workunit.client.0.smithi116.stderr:7' 2024-06-15T09:46:05.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:46:05.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:05.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:46:05.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-15T09:46:05.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-15T09:46:05.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-15T09:46:05.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:05.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:46:05.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-06-15T09:46:05.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-06-15T09:46:05.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2024-06-15T09:46:05.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:05.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:46:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-06-15T09:46:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-06-15T09:46:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454' 2024-06-15T09:46:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:05.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T09:46:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-06-15T09:46:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-06-15T09:46:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345932' 2024-06-15T09:46:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:06.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T09:46:06.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182411 2024-06-15T09:46:06.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182411 2024-06-15T09:46:06.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345932 4-107374182411' 2024-06-15T09:46:06.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:06.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T09:46:06.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-06-15T09:46:06.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-06-15T09:46:06.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345932 4-107374182411 5-128849018887' 2024-06-15T09:46:06.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:06.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T09:46:06.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-06-15T09:46:06.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-06-15T09:46:06.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345932 4-107374182411 5-128849018887 6-150323855365' 2024-06-15T09:46:06.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:46:06.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T09:46:06.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T09:46:06.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T09:46:06.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345932 4-107374182411 5-128849018887 6-150323855365 7-171798691842' 2024-06-15T09:46:06.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:06.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-15T09:46:06.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:06.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:46:06.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-15T09:46:06.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:06.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-15T09:46:06.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-15T09:46:06.822 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836499 2024-06-15T09:46:06.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:46:07.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-15T09:46:07.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:07.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-06-15T09:46:07.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:07.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:46:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-06-15T09:46:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:07.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-06-15T09:46:07.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-15T09:46:07.058 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672977 2024-06-15T09:46:07.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:46:07.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-06-15T09:46:07.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:07.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-06-15T09:46:07.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:07.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:46:07.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-06-15T09:46:07.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:07.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-06-15T09:46:07.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-15T09:46:07.335 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509454 2024-06-15T09:46:07.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:46:07.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509454 2024-06-15T09:46:07.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:07.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-06-15T09:46:07.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:07.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T09:46:07.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-06-15T09:46:07.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:07.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-06-15T09:46:07.602 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345932 2024-06-15T09:46:07.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-06-15T09:46:07.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T09:46:07.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-06-15T09:46:07.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:07.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182411 2024-06-15T09:46:07.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:07.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T09:46:07.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182411 2024-06-15T09:46:07.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:07.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182411 2024-06-15T09:46:07.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182411' 2024-06-15T09:46:07.859 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182411 2024-06-15T09:46:07.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T09:46:08.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182411 -lt 107374182411 2024-06-15T09:46:08.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:08.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-06-15T09:46:08.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:08.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T09:46:08.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-06-15T09:46:08.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:08.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-06-15T09:46:08.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-06-15T09:46:08.105 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.5 seq 128849018887 2024-06-15T09:46:08.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T09:46:08.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-06-15T09:46:08.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:08.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-06-15T09:46:08.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:08.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T09:46:08.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-06-15T09:46:08.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-06-15T09:46:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-06-15T09:46:08.353 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.6 seq 150323855365 2024-06-15T09:46:08.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T09:46:08.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-06-15T09:46:08.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:46:09.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:46:09.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T09:46:09.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-15T09:46:09.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:46:09.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-06-15T09:46:09.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:46:09.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T09:46:09.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-06-15T09:46:09.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:46:09.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T09:46:09.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-06-15T09:46:09.827 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.7 seq 171798691842 2024-06-15T09:46:09.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T09:46:10.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-06-15T09:46:10.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:46:10.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:46:10.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:46:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-06-15T09:46:10.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:46:10.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:46:10.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:46:10.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:46:10.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:46:10.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:46:10.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:46:10.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-06-15T09:46:10.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:46:10.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:46:10.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:46:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-06-15T09:46:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:46:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:46:10.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-06-15T09:46:10.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-06-15T09:46:10.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-06-15T09:46:11.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-06-15T09:46:11.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.37079 bs=1M count=50 2024-06-15T09:46:11.318 INFO:tasks.workunit.client.0.smithi116.stderr:50+0 records in 2024-06-15T09:46:11.318 INFO:tasks.workunit.client.0.smithi116.stderr:50+0 records out 2024-06-15T09:46:11.318 INFO:tasks.workunit.client.0.smithi116.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.168172 s, 312 MB/s 2024-06-15T09:46:11.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-06-15T09:46:11.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:11.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.37079 2024-06-15T09:46:13.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:13.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.37079 2024-06-15T09:46:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.37079 2024-06-15T09:46:16.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:16.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.37079 2024-06-15T09:46:18.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.37079 2024-06-15T09:46:20.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:20.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.37079 2024-06-15T09:46:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:21.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.37079 2024-06-15T09:46:23.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:23.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.37079 2024-06-15T09:46:25.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:25.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.37079 2024-06-15T09:46:26.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:26.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.37079 2024-06-15T09:46:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:28.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.37079 2024-06-15T09:46:30.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:30.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.37079 2024-06-15T09:46:32.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:32.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.37079 2024-06-15T09:46:34.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:34.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.37079 2024-06-15T09:46:36.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:36.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.37079 2024-06-15T09:46:37.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:37.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.37079 2024-06-15T09:46:39.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:39.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.37079 2024-06-15T09:46:41.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:41.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.37079 2024-06-15T09:46:43.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:43.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.37079 2024-06-15T09:46:45.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:45.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.37079 2024-06-15T09:46:47.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:47.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.37079 2024-06-15T09:46:48.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:48.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.37079 2024-06-15T09:46:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.37079 2024-06-15T09:46:52.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:52.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.37079 2024-06-15T09:46:54.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:54.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.37079 2024-06-15T09:46:56.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:56.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.37079 2024-06-15T09:46:57.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:57.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.37079 2024-06-15T09:46:59.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:46:59.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.37079 2024-06-15T09:47:01.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:01.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.37079 2024-06-15T09:47:03.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:03.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.37079 2024-06-15T09:47:05.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:05.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.37079 2024-06-15T09:47:06.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:06.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.37079 2024-06-15T09:47:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:08.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.37079 2024-06-15T09:47:10.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:10.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.37079 2024-06-15T09:47:12.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:12.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.37079 2024-06-15T09:47:13.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:13.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.37079 2024-06-15T09:47:15.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:15.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.37079 2024-06-15T09:47:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.37079 2024-06-15T09:47:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.37079 2024-06-15T09:47:20.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-06-15T09:47:20.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.37079 2024-06-15T09:47:22.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.37079 2024-06-15T09:47:22.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-06-15T09:47:23.008 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 3 2024-06-15T09:47:23.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-15T09:47:23.248 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:47:23.248 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-15T09:46:05.483971+0000 47'26 47:58 [5] 5 [5] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.799266+0000 47'13 47:44 [3] 3 [3] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.759287+0000 47'13 47:33 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.576729+0000 47'13 47:36 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.758058+0000 47'13 47:36 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:04.576790+0000 0'0 47:23 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-15T09:46:05.483110+0000 47'26 47:48 [5] 5 [5] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.482996+0000 47'13 47:34 [6] 6 [6] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 1 0 0 0 0 37748736 0 0 9 0 9 active+clean 2024-06-15T09:46:05.483028+0000 47'9 47:32 [6] 6 [6] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.249 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-15T09:46:05.878553+0000 47'39 47:60 [4] 4 [4] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-15T09:46:04.981251+0000 47'52 47:75 [3] 3 [3] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-06-15T09:46:05.998044+0000 47'52 47:73 [4] 4 [4] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.851357+0000 47'13 47:39 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:04.798958+0000 0'0 47:20 [3] 3 [3] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.349000+0000 47'13 47:45 [5] 5 [5] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.027545+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.851891+0000 47'13 47:43 [7] 7 [7] 7 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.851390+0000 47'13 47:32 [7] 7 [7] 7 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.758467+0000 47'13 47:34 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.374201+0000 47'13 47:32 [6] 6 [6] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.250 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.483868+0000 47'13 47:33 [5] 5 [5] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.852361+0000 47'13 47:33 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-06-15T09:46:04.851570+0000 47'26 47:46 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.998084+0000 47'13 47:30 [4] 4 [4] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:05.852579+0000 0'0 47:18 [2] 2 [2] 2 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-06-15T09:46:04.758304+0000 47'39 47:60 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:05.483376+0000 0'0 47:19 [6] 6 [6] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.027269+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:05.482972+0000 0'0 47:19 [5] 5 [5] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:04.798955+0000 47'13 47:31 [3] 3 [3] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:46:04.758511+0000 0'0 47:20 [1] 1 [1] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-06-15T09:46:05.026701+0000 47'13 47:32 [0] 0 [0] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:47:23.251 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:47:23.252 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:47:23.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-06-15T09:47:23.464 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:47:23.464 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_recovery_max_active": "0" 2024-06-15T09:47:23.464 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:47:23.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-06-15T09:47:23.809 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:47:23.809 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_recovery_max_active_hdd": "3" 2024-06-15T09:47:23.810 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:47:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-06-15T09:47:24.114 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:47:24.114 INFO:tasks.workunit.client.0.smithi116.stdout: "osd_recovery_max_active_ssd": "10" 2024-06-15T09:47:24.114 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:47:24.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-06-15T09:47:24.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-15T09:47:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-15T09:47:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-15T09:47:24.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-15T09:47:24.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-15T09:47:24.435 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:47:24.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-06-15T09:47:24.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-15T09:47:26.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-06-15T09:47:26.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-06-15T09:47:26.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-06-15T09:47:26.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-15T09:47:26.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-15T09:47:26.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-15T09:47:26.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-15T09:47:26.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-15T09:47:26.687 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:47:26.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-06-15T09:47:26.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-06-15T09:47:28.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-06-15T09:47:28.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-06-15T09:47:28.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-06-15T09:47:28.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-06-15T09:47:28.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-06-15T09:47:28.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-06-15T09:47:28.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-06-15T09:47:28.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-06-15T09:47:28.935 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:47:28.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-06-15T09:47:28.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-06-15T09:47:28.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-06-15T09:47:29.166 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-15T09:47:25.233186+0000 47'26 50:68 [5,0,2] 5 [5,0,2] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.677498+0000 47'13 50:54 [3,0,7] 3 [3,0,7] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 activating 2024-06-15T09:47:24.423274+0000 47'26 50:50 [1,3,0] 1 [1,3,0] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-06-15T09:47:24.419191+0000 47'13 50:40 [1,0,2] 1 [1,0,2] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-06-15T09:47:24.416494+0000 47'13 50:40 [1,6,2] 1 [1,6,2] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 activating 2024-06-15T09:47:24.425746+0000 0'0 50:27 [1,4,5] 1 [1,4,5] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-06-15T09:47:25.675885+0000 47'26 50:57 [5,4,3] 5 [5,4,3] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-06-15T09:47:25.227808+0000 47'13 50:39 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.449389+0000 47'13 50:46 [6,7,3] 6 [6,7,3] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 activating+degraded 2024-06-15T09:47:25.012789+0000 47'39 50:65 [4,2,6] 4 [4,2,6] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-06-15T09:47:25.122624+0000 47'52 50:85 [3,0,6] 3 [3,0,6] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.167 INFO:tasks.workunit.client.0.smithi116.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 peering 2024-06-15T09:47:23.284136+0000 47'52 49:76 [4,1,2] 4 [4,1,2] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.678344+0000 47'13 50:48 [0,7,1] 0 [0,7,1] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:47:24.879172+0000 0'0 50:31 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.342721+0000 47'13 50:55 [5,0,4] 5 [5,0,4] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.234861+0000 47'13 50:41 [0,3,4] 0 [0,3,4] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-06-15T09:47:25.226878+0000 47'13 50:56 [7,0,2] 7 [7,0,2] 7 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.012910+0000 47'13 50:43 [7,4,2] 7 [7,4,2] 7 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-06-15T09:47:24.659967+0000 47'13 50:39 [1,4,2] 1 [1,4,2] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.017080+0000 47'13 50:42 [6,0,3] 6 [6,0,3] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.676323+0000 47'13 50:43 [5,7,6] 5 [5,7,6] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.340435+0000 47'13 50:42 [0,1,3] 0 [0,1,3] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovering+degraded 2024-06-15T09:47:25.141174+0000 47'26 50:56 [0,1,6] 0 [0,1,6] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.168 INFO:tasks.workunit.client.0.smithi116.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-06-15T09:47:24.771030+0000 47'13 50:34 [4,6,3] 4 [4,6,3] 4 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:47:25.342178+0000 0'0 50:29 [2,0,4] 2 [2,0,4] 2 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 activating 2024-06-15T09:47:24.417621+0000 47'39 50:64 [1,5,3] 1 [1,5,3] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:47:25.452052+0000 0'0 50:29 [6,0,7] 6 [6,0,7] 6 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.450265+0000 47'13 50:41 [0,4,5] 0 [0,4,5] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-06-15T09:47:25.015565+0000 0'0 50:30 [5,2,3] 5 [5,2,3] 5 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 periodic scrub scheduled @ 2024-06-16T09:46:03.466858+0000 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.340202+0000 47'13 50:40 [3,0,4] 3 [3,0,4] 3 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 activating 2024-06-15T09:47:24.415726+0000 0'0 50:24 [1,7,4] 1 [1,7,4] 1 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-06-15T09:47:25.229824+0000 47'13 50:41 [0,1,2] 0 [0,1,2] 0 0'0 2024-06-15T09:46:03.466858+0000 0'0 2024-06-15T09:46:03.466858+0000 0 0 no scrub is scheduled 0 0 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:47:29.169 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:47:29.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-06-15T09:47:29.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-06-15T09:47:29.182 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-06-15T09:47:29.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-06-15T09:47:29.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-06-15T09:47:29.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-06-15T09:47:29.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58630"' 2024-06-15T09:47:29.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58630' 2024-06-15T09:47:29.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58633: /' 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58637"' 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58637' 2024-06-15T09:47:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-06-15T09:47:29.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-06-15T09:47:29.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58640: /' 2024-06-15T09:47:29.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58642"' 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58642' 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58647: /' 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58649"' 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58649' 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58654: /' 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-06-15T09:47:29.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58656"' 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58656' 2024-06-15T09:47:29.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-06-15T09:47:29.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-06-15T09:47:29.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58662"' 2024-06-15T09:47:29.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58662' 2024-06-15T09:47:29.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-06-15T09:47:29.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58661: /' 2024-06-15T09:47:29.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-06-15T09:47:29.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-06-15T09:47:29.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58667"' 2024-06-15T09:47:29.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58667' 2024-06-15T09:47:29.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58671"' 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58671' 2024-06-15T09:47:29.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-06-15T09:47:29.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-06-15T09:47:29.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58675"' 2024-06-15T09:47:29.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58675' 2024-06-15T09:47:29.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-06-15T09:47:29.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-06-15T09:47:29.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-06-15T09:47:29.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58680"' 2024-06-15T09:47:29.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58680' 2024-06-15T09:47:29.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-06-15T09:47:29.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58666: /' 2024-06-15T09:47:29.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58683: /' 2024-06-15T09:47:29.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58674: /' 2024-06-15T09:47:29.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-06-15T09:47:29.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-06-15T09:47:29.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58685"' 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58685' 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-06-15T09:47:29.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58690"' 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58690' 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-06-15T09:47:29.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58698: /' 2024-06-15T09:47:29.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58697: /' 2024-06-15T09:47:29.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-06-15T09:47:29.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58699: /' 2024-06-15T09:47:29.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-06-15T09:47:29.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58705"' 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58705' 2024-06-15T09:47:29.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-06-15T09:47:29.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58712"' 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58712' 2024-06-15T09:47:29.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58688: /' 2024-06-15T09:47:29.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58717: /' 2024-06-15T09:47:29.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-06-15T09:47:29.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58721"' 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58721' 2024-06-15T09:47:29.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-06-15T09:47:29.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-06-15T09:47:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58728: /' 2024-06-15T09:47:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-06-15T09:47:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58729"' 2024-06-15T09:47:29.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58729' 2024-06-15T09:47:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58724: /' 2024-06-15T09:47:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-06-15T09:47:29.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58735"' 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58735' 2024-06-15T09:47:29.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58740: /' 2024-06-15T09:47:29.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-06-15T09:47:29.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-06-15T09:47:29.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-06-15T09:47:29.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58744"' 2024-06-15T09:47:29.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58744' 2024-06-15T09:47:29.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58738: /' 2024-06-15T09:47:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-06-15T09:47:29.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-06-15T09:47:29.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-06-15T09:47:29.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58751"' 2024-06-15T09:47:29.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58751' 2024-06-15T09:47:29.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-06-15T09:47:29.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-06-15T09:47:29.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58753"' 2024-06-15T09:47:29.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58753' 2024-06-15T09:47:29.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58750: /' 2024-06-15T09:47:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-06-15T09:47:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58759: /' 2024-06-15T09:47:29.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58764"' 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58764' 2024-06-15T09:47:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-06-15T09:47:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58761: /' 2024-06-15T09:47:29.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-06-15T09:47:29.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-06-15T09:47:29.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-06-15T09:47:29.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58771"' 2024-06-15T09:47:29.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58771' 2024-06-15T09:47:29.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58770: /' 2024-06-15T09:47:29.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-06-15T09:47:29.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58778"' 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58778' 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58775: /' 2024-06-15T09:47:29.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58782"' 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58782' 2024-06-15T09:47:29.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-06-15T09:47:29.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58790: /' 2024-06-15T09:47:29.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-06-15T09:47:29.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58795"' 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58795' 2024-06-15T09:47:29.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58788: /' 2024-06-15T09:47:29.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-06-15T09:47:29.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58803: /' 2024-06-15T09:47:29.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58804"' 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58804' 2024-06-15T09:47:29.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-06-15T09:47:29.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-06-15T09:47:29.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58809"' 2024-06-15T09:47:29.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58809' 2024-06-15T09:47:29.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58808: /' 2024-06-15T09:47:29.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-06-15T09:47:29.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58821"' 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58821' 2024-06-15T09:47:29.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58819: /' 2024-06-15T09:47:29.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-06-15T09:47:29.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-06-15T09:47:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-06-15T09:47:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58832"' 2024-06-15T09:47:29.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58832' 2024-06-15T09:47:29.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-06-15T09:47:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58839"' 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58839' 2024-06-15T09:47:29.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58831: /' 2024-06-15T09:47:29.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-06-15T09:47:29.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58837: /' 2024-06-15T09:47:29.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58857"' 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58857' 2024-06-15T09:47:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-06-15T09:47:29.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-06-15T09:47:29.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58856: /' 2024-06-15T09:47:29.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58862"' 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58862' 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 58630 58637 58642 58649 58656 58662 58667 58671 58675 58680 58685 58690 58705 58712 58721 58729 58735 58744 58751 58753 58764 58771 58778 58782 58795 58804 58809 58821 58832 58839 58857 58862' 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58630 2024-06-15T09:47:29.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58861: /' 2024-06-15T09:47:29.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-06-15T09:47:29.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-06-15T09:47:29.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58867: /' 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: instructing pg 1.0 on osd.7 to scrub 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-06-15T09:47:35.037 INFO:tasks.workunit.client.0.smithi116.stderr:58633: dumped pgs 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:47:35.038 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.039 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.112 INFO:tasks.workunit.client.0.smithi116.stderr:58633: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-06-15T09:47:35.112 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-06-15T09:47:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:58654: instructing pg 1.3 on osd.4 to scrub 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: dumped pgs 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:58654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.205 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: instructing pg 1.15 on osd.1 to scrub 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: dumped pgs 2024-06-15T09:47:35.206 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.207 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.340 INFO:tasks.workunit.client.0.smithi116.stderr:58775: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stat58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-06-15T09:47:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-06-15T09:47:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: instructing pg 1.12 on osd.0 to scrub 2024-06-15T09:47:35.408 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: dumped pgs 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.409 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-06-15T09:47:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-06-15T09:47:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.410 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.711 INFO:tasks.workunit.client.0.smithi116.stderr:58761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stat58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-06-15T09:47:35.711 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-06-15T09:47:35.711 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-15T09:47:35.711 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.711 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: instructing pg 1.7 on osd.6 to scrub 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.712 INFO:tasks.workunit.client.0.smithi116.stderr:58699: dumped pgs 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.713 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-06-15T09:47:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:58699: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:58699: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.732 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: instructing pg 1.c on osd.3 to scrub 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: dumped pgs 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.733 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.734 INFO:tasks.workunit.client.0.smithi116.stderr:58717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-06-15T09:47:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:47:32.527897+0000 '>' '' 2024-06-15T09:47:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:58717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.811 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: instructing pg 1.16 on osd.6 to scrub 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: dumped pgs 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.812 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.813 INFO:tasks.workunit.client.0.smithi116.stderr:58788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.861 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.862 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: instructing pg 1.19 on osd.3 to scrub 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: dumped pgs 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:58808: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.880 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-06-15T09:47:35.880 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: instructing pg 1.11 on osd.0 to scrub 2024-06-15T09:47:35.881 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: dumped pgs 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.882 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:58750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.898 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-06-15T09:47:35.898 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: instructing pg 1.17 on osd.0 to scrub 2024-06-15T09:47:35.899 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: dumped pgs 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.900 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:58790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:35.957 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-06-15T09:47:35.957 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-06-15T09:47:35.957 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-15T09:47:35.957 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.957 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: instructing pg 1.2 on osd.0 to scrub 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:35.958 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: dumped pgs 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-06-15T09:47:35.959 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:58647: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:58647: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.008 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-06-15T09:47:36.008 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: instructing pg 1.8 on osd.5 to scrub 2024-06-15T09:47:36.009 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: dumped pgs 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-06-15T09:47:36.010 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:58698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-15T09:47:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: instructing pg 1.1f on osd.5 to scrub 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: dumped pgs 2024-06-15T09:47:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:58867: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.034 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-06-15T09:47:36.034 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-06-15T09:47:36.034 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.035 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: instructing pg 1.1 on osd.5 to scrub 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: dumped pgs 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:58640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.059 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.060 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: instructing pg 1.1a on osd.1 to scrub 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: dumped pgs 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.061 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:58819: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.064 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-15T09:47:36.065 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.066 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.066 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-06-15T09:47:36.066 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.066 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.066 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.067 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-06-15T09:47:36.067 INFO:tasks.workunit.client.0.smithi116.stderr:58674: instructing pg 1.6 on osd.6 to scrub 2024-06-15T09:47:36.067 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.067 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-06-15T09:47:36.068 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.068 INFO:tasks.workunit.client.0.smithi116.stderr:58674: dumped pgs 2024-06-15T09:47:36.068 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.068 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-06-15T09:47:36.068 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.069 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-06-15T09:47:36.070 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-06-15T09:47:36.070 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.070 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.070 INFO:tasks.workunit.client.0.smithi116.stderr:58674: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-06-15T09:47:36.070 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.071 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.071 INFO:tasks.workunit.client.0.smithi116.stderr:58674: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.073 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-06-15T09:47:36.073 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-06-15T09:47:36.073 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-15T09:47:36.073 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.073 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.074 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-06-15T09:47:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:58661: instructing pg 1.4 on osd.3 to scrub 2024-06-15T09:47:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.075 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: dumped pgs 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.076 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-06-15T09:47:36.077 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-06-15T09:47:36.078 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.078 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-06-15T09:47:36.078 INFO:tasks.workunit.client.0.smithi116.stderr:58661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.078 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.078 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.079 INFO:tasks.workunit.client.0.smithi116.stderr:58661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.164 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.165 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: instructing pg 1.5 on osd.4 to scrub 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: dumped pgs 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.166 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:58666: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.168 INFO:tasks.workunit.client.0.smithi116.stderr:lect(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:47:36.168 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.168 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.168 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-15T09:47:36.168 INFO:tasks.workunit.client.0.smithi116.stderr:58633: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-15T09:47:36.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-06-15T09:47:36.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-15T09:47:36.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-15T09:47:36.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58637 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58642 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58649 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58656 2024-06-15T09:47:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58662 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58667 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58671 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58675 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58680 2024-06-15T09:47:36.187 INFO:tasks.workunit.client.0.smithi116.stderr:s | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-06-15T09:47:36.187 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.188 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.188 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-15T09:47:36.188 INFO:tasks.workunit.client.0.smithi116.stderr:58761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-15T09:47:36.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.231 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: instructing pg 1.a on osd.1 to scrub 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: dumped pgs 2024-06-15T09:47:36.232 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-06-15T09:47:36.233 INFO:tasks.workunit.client.0.smithi116.stderr:58688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.234 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.234 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.234 INFO:tasks.workunit.client.0.smithi116.stderr:58688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.240 INFO:tasks.workunit.client.0.smithi116.stderr:s | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-06-15T09:47:36.240 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.240 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.240 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-06-15T09:47:36.240 INFO:tasks.workunit.client.0.smithi116.stderr:58775: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-06-15T09:47:36.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-06-15T09:47:36.796 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.797 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: instructing pg 1.f on osd.6 to scrub 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: dumped pgs 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.798 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:58738: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.807 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: instructing pg 1.18 on osd.5 to scrub 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: dumped pgs 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.808 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:58803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.837 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: instructing pg 1.d on osd.7 to scrub 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: dumped pgs 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.838 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:58724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.850 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: instructing pg 1.13 on osd.4 to scrub 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: dumped pgs 2024-06-15T09:47:36.851 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.852 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58759: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-06-15T09:47:36.853 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: instructing pg 1.1e on osd.3 to scrub 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: dumped pgs 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:58861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.864 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: instructing pg 1.1d on osd.1 to scrub 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: dumped pgs 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.865 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.866 INFO:tasks.workunit.client.0.smithi116.stderr:58856: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.884 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: instructing pg 1.1b on osd.0 to scrub 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: dumped pgs 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.885 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:58831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-15T09:47:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: instructing pg 1.e on osd.1 to scrub 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: dumped pgs 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.889 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.890 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.891 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.891 INFO:tasks.workunit.client.0.smithi116.stderr:58728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.903 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: instructing pg 1.9 on osd.1 to scrub 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: dumped pgs 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.904 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:47:33.874153+0000 '>' '' 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.905 INFO:tasks.workunit.client.0.smithi116.stderr:58683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58685 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58690 2024-06-15T09:47:36.919 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.920 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: instructing pg 1.14 on osd.2 to scrub 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: dumped pgs 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:47:34.032436+0000 '>' '' 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:58770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: instructing pg 1.1c on osd.0 to scrub 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: dumped pgs 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.930 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.932 INFO:tasks.workunit.client.0.smithi116.stderr:58837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.937 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: instructing pg 1.b on osd.1 to scrub 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: dumped pgs 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-06-15T09:47:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:58697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58705 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58712 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58721 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58729 2024-06-15T09:47:36.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58735 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-06-15T09:47:36.952 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: instructing pg 1.10 on osd.5 to scrub 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: dumped pgs 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-06-15T09:46:03.466858+0000 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-06-15T09:47:36.953 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-06-15T09:46:03.466858+0000 '>' '' 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:58740: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58744 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58751 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58753 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58764 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58771 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 2 + 1 2024-06-15T09:47:36.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=3 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58778 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58782 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58795 2024-06-15T09:47:36.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58804 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58809 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58821 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58832 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58839 2024-06-15T09:47:36.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58857 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58862 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-06-15T09:47:36.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 3 -eq 0 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.2.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.3.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.7.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.4.pid 2024-06-15T09:47:36.961 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/mon.a.pid 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.1.pid 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.0.pid 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53644 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:49511 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:50890 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:45846 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:56444 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:52268 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:45762 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:48135 2024-06-15T09:47:36.962 INFO:tasks.workunit.client.0.smithi116.stderr:46759 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:55044' 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53644 49511 50890 45846 56444 52268 45762 48135 46759 55044 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:47:36.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:47:39.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:47:39.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-06-15T09:47:39.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-06-15T09:47:39.264 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-06-15T09:47:39.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-06-15T09:47:39.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-06-15T09:47:39.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-06-15T09:47:39.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-06-15T09:47:39.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-06-15T09:47:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-06-15T09:47:39.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-06-15T09:47:39.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-06-15T09:47:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-06-15T09:47:39.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-06-15T09:47:39.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-06-15T09:47:39.304 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-06-15T09:47:39.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-06-15T09:47:39.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-06-15T09:47:39.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-06-15T09:47:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-06-15T09:47:39.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-06-15T09:47:39.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-06-15T09:47:39.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-06-15T09:47:39.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-06-15T09:47:39.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T09:47:39.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-06-15T09:47:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-06-15T09:47:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-06-15T09:47:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-15T09:47:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:47:39.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-15T09:47:39.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:47:39.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:47:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:47:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:47:39.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:47:39.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:47:39.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:47:39.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:47:39.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:47:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:47:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:47:39.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:47:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:47:39.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:47:39.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:47:39.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:47:39.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-15T09:47:39.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:47:39.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:39.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:47:39.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37079 2024-06-15T09:47:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:47:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:47:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-06-15T09:47:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-06-15T09:47:39.385 INFO:tasks.workunit.client.0.smithi116.stdout:TEST PASSED 2024-06-15T09:47:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-06-15T09:47:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:47:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:47:39.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:47:39.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:47:39.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:47:39.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:47:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:47:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:47:39.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:47:39.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:47:39.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:47:39.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:47:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-15T09:47:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-06-15T09:47:39.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:47:39.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:39.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.37079 2024-06-15T09:47:39.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.37079 2024-06-15T09:47:39.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:47:39.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:47:39.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-15T09:47:39.396 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-15T09:47:39.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-15T09:47:39.448 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-06-15T09:47:39.449 DEBUG:teuthology.orchestra.run.smithi116:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-06-15T09:47:39.501 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-15T09:47:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-15T09:47:39.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-06-15T09:47:39.507 INFO:tasks.workunit.client.0.smithi116.stdout:This test is disabled 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-06-15T09:47:39.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:47:39.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:47:39.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:47:39.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:47:39.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:47:39.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:47:39.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:47:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:47:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:47:39.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:47:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-06-15T09:47:39.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-06-15T09:47:39.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:47:39.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:39.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62069 2024-06-15T09:47:39.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62069 2024-06-15T09:47:39.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:47:39.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:47:39.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-06-15T09:47:39.517 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-15T09:47:39.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-15T09:47:39.568 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-06-15T09:47:39.568 DEBUG:teuthology.orchestra.run.smithi116:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-06-15T09:47:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-06-15T09:47:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:++ TIMEOUT=300 2024-06-15T09:47:39.621 INFO:tasks.workunit.client.0.smithi116.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-06-15T09:47:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:++ MAX_TIMEOUT=15 2024-06-15T09:47:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:++ PG_NUM=4 2024-06-15T09:47:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:++ TMPDIR=/tmp 2024-06-15T09:47:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-06-15T09:47:39.622 INFO:tasks.workunit.client.0.smithi116.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-06-15T09:47:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:++ type xmlstarlet 2024-06-15T09:47:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:++ XMLSTARLET=xmlstarlet 2024-06-15T09:47:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:+++ uname 2024-06-15T09:47:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' Linux = FreeBSD ']' 2024-06-15T09:47:39.624 INFO:tasks.workunit.client.0.smithi116.stderr:++ SED=sed 2024-06-15T09:47:39.624 INFO:tasks.workunit.client.0.smithi116.stderr:++ AWK=awk 2024-06-15T09:47:39.624 INFO:tasks.workunit.client.0.smithi116.stderr:+++ stty -a 2024-06-15T09:47:39.624 INFO:tasks.workunit.client.0.smithi116.stderr:+++ head -1 2024-06-15T09:47:39.625 INFO:tasks.workunit.client.0.smithi116.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-06-15T09:47:39.625 INFO:tasks.workunit.client.0.smithi116.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ termwidth= 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ DIFFCOLOPTS='-y ' 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ KERNCORE=kernel.core_pattern 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ EXTRA_OPTS= 2024-06-15T09:47:39.628 INFO:tasks.workunit.client.0.smithi116.stderr:++ test '' = TESTS 2024-06-15T09:47:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-06-15T09:47:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:++ uname 2024-06-15T09:47:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' Linux = FreeBSD ']' 2024-06-15T09:47:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:+ use_ec_overwrite=true 2024-06-15T09:47:39.629 INFO:tasks.workunit.client.0.smithi116.stderr:+ getjson=no 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr:+ jqfilter='def walk(f): 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T09:47:39.630 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T09:47:39.631 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T09:47:39.631 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T09:47:39.631 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T09:47:39.631 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T09:47:39.631 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T09:47:39.632 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T09:47:39.632 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T09:47:39.632 INFO:tasks.workunit.client.0.smithi116.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T09:47:39.635 INFO:tasks.workunit.client.0.smithi116.stderr:+ main osd-scrub-repair 2024-06-15T09:47:39.635 INFO:tasks.workunit.client.0.smithi116.stderr:+ local dir=td/osd-scrub-repair 2024-06-15T09:47:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:+ shift 2024-06-15T09:47:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:+ shopt -s -o xtrace 2024-06-15T09:47:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-06-15T09:47:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:47:39.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-06-15T09:47:39.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-06-15T09:47:39.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-06-15T09:47:39.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-06-15T09:47:39.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-06-15T09:47:39.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-06-15T09:47:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-06-15T09:47:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-06-15T09:47:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-06-15T09:47:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-06-15T09:47:39.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-06-15T09:47:39.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-06-15T09:47:39.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-06-15T09:47:39.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-06-15T09:47:39.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none ' 2024-06-15T09:47:39.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-06-15T09:47:39.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-06-15T09:47:39.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-06-15T09:47:39.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-06-15T09:47:39.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-06-15T09:47:39.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-06-15T09:47:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_bluestore_basic 2024-06-15T09:47:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_bluestore_failed 2024-06-15T09:47:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-06-15T09:47:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_bluestore_scrub 2024-06-15T09:47:39.645 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_bluestore_tag 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_erasure_coded_appends 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_and_repair_lrc_appends 2024-06-15T09:47:39.646 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-06-15T09:47:39.647 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_and_repair_replicated 2024-06-15T09:47:39.647 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_scrub_erasure_appends 2024-06-15T09:47:39.647 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-06-15T09:47:39.647 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_scrub_replicated 2024-06-15T09:47:39.647 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_corrupt_snapset_scrub_rep 2024-06-15T09:47:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_list_missing_erasure_coded_appends 2024-06-15T09:47:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_list_missing_erasure_coded_overwrites 2024-06-15T09:47:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_periodic_scrub_replicated 2024-06-15T09:47:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_repair_stats 2024-06-15T09:47:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_repair_stats_ec 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_request_scrub_priority 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_scrub_warning 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_skip_non_repair_during_recovery 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_unfound_erasure_coded_appends 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:TEST_unfound_erasure_coded_overwrites' 2024-06-15T09:47:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T09:47:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T09:47:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T09:47:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T09:47:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:47:39.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:47:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:47:39.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:47:39.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:47:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:47:39.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:47:39.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:47:39.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:47:39.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:47:39.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:47:39.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:47:39.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:47:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:47:39.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T09:47:39.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T09:47:39.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-15T09:47:40.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:47:40.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:47:40.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:47:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:47:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:40.065 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:40.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:47:40.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-15T09:47:40.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:47:40.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:47:40.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:47:40.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:47:40.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:47:40.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:47:40.303 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:40.304 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:47:40.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:47:40.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T09:47:40.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:47:40.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:47:40.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:47:40.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:47:40.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:47:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:47:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:47:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:47:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:47:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:40.423 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:47:40.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:47:40.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T09:47:40.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-06-15T09:47:40.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T09:47:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:47:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:47:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:47:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T09:47:40.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:47:41.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:47:41.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:47:41.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:47:41.399 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:47:41.399 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:41.400 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:41.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:47:41.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:47:41.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:47:41.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-15T09:47:41.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:47:41.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:47:41.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:47:41.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:47:41.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:47:41.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:47:41.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88d4c734-5576-44ce-9cd7-0ad897c2326b 2024-06-15T09:47:41.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 88d4c734-5576-44ce-9cd7-0ad897c2326b' 2024-06-15T09:47:41.441 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 88d4c734-5576-44ce-9cd7-0ad897c2326b 2024-06-15T09:47:41.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:47:41.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9Y21mNJGIGxAAIzW3AD9TqN/QNPO93NRYGw== 2024-06-15T09:47:41.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9Y21mNJGIGxAAIzW3AD9TqN/QNPO93NRYGw=="}' 2024-06-15T09:47:41.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88d4c734-5576-44ce-9cd7-0ad897c2326b -i td/osd-scrub-repair/0/new.json 2024-06-15T09:47:41.745 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:47:41.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T09:47:41.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQA9Y21mNJGIGxAAIzW3AD9TqN/QNPO93NRYGw== --osd-uuid 88d4c734-5576-44ce-9cd7-0ad897c2326b 2024-06-15T09:47:41.798 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.799+0000 7fade4c877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:41.812 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.811+0000 7fade4c877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:41.813 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.815+0000 7fade4c877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:41.814 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.815+0000 7fade4c877c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:47:41.814 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.815+0000 7fade4c877c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:47:41.814 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.815+0000 7fade4c877c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:47:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:41.911+0000 7fade4c877c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T09:47:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T09:47:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:47:48.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:47:48.137 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:47:48.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:47:48.397 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:47:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:47:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:47:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:47:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-15T09:47:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:47:48.430 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:48.431+0000 7fb6ab7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:48.433 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:48.431+0000 7fb6ab7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:48.434 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:48.435+0000 7fb6ab7067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:47:48.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:47:48.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:47:48.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:47:48.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:47:48.668 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:47:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:47:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:47:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:48.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:48.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:48.992 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:48.991+0000 7fb6ab7067c0 -1 Falling back to public interface 2024-06-15T09:47:49.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:49.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:49.920 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:47:49.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:47:49.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:49.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:50.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:51.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:51.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:51.158 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:47:51.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:47:51.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:51.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:51.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:51.677 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:47:51.675+0000 7fb6ab7067c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:47:52.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:52.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:52.401 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:47:52.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:47:52.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:52.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:52.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:53.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:53.756 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:47:53.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:53.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:47:53.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:53.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:53.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:54.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:54.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:54.997 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:47:54.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:47:54.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:54.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:55.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:56.241 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:47:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:47:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:56.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:57.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:57.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:57.597 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:47:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:47:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:57.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:57.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:47:58.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:47:58.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:47:58.838 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:47:58.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:47:58.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:47:58.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:47:59.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:00.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:00.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:00.082 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:48:00.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:48:00.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:00.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:00.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:01.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:01.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:01.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:48:01.336 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:48:01.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:01.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:01.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:02.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:02.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:02.758 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:48:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:48:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:03.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:04.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:04.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:04.002 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:48:04.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:48:04.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:04.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:05.240 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:48:05.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:05.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:48:05.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:05.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:06.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:06.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:06.697 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:48:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:48:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:06.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:06.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:07.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:07.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:07.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:48:07.942 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:48:07.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:07.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:08.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:09.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:09.342 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:48:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:48:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:09.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:10.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:10.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:10.997 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:48:10.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:48:10.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:10.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:11.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:12.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:12.554 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:48:12.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:12.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:48:12.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:12.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:13.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:14.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:14.331 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:48:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:48:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:14.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:15.574 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:48:15.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:48:15.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:15.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:16.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:17.022 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:48:17.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:48:17.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:17.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:48:17.278 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3625881450,v1:127.0.0.1:6803/3625881450] [v2:127.0.0.1:6804/3625881450,v1:127.0.0.1:6805/3625881450] exists,up 88d4c734-5576-44ce-9cd7-0ad897c2326b 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:48:17.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:48:17.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:48:17.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:48:17.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:48:17.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4805f693-8cf9-4934-ae11-beade780bd92 2024-06-15T09:48:17.287 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 4805f693-8cf9-4934-ae11-beade780bd92 2024-06-15T09:48:17.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4805f693-8cf9-4934-ae11-beade780bd92' 2024-06-15T09:48:17.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:48:17.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhY21mIdhOEhAAO+lvcmiEkB1tCFOzDTwtcg== 2024-06-15T09:48:17.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhY21mIdhOEhAAO+lvcmiEkB1tCFOzDTwtcg=="}' 2024-06-15T09:48:17.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4805f693-8cf9-4934-ae11-beade780bd92 -i td/osd-scrub-repair/1/new.json 2024-06-15T09:48:17.604 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:48:17.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T09:48:17.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBhY21mIdhOEhAAO+lvcmiEkB1tCFOzDTwtcg== --osd-uuid 4805f693-8cf9-4934-ae11-beade780bd92 2024-06-15T09:48:17.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.659+0000 7ff83fff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.699+0000 7ff83fff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:17.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.699+0000 7ff83fff47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:17.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.699+0000 7ff83fff47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:48:17.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.699+0000 7ff83fff47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:48:17.701 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.699+0000 7ff83fff47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:48:17.790 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:17.791+0000 7ff83fff47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T09:48:25.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T09:48:25.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:48:25.150 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:48:25.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:48:25.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:48:25.415 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:48:25.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:48:25.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-06-15T09:48:25.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:48:25.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:48:25.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:48:25.451 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:25.451+0000 7effbddd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:25.478 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:25.479+0000 7effbddd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:25.479 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:25.479+0000 7effbddd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:48:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:48:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:48:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:48:25.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:48:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:48:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:48:25.659 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:48:25.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:25.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:25.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:26.031+0000 7effbddd17c0 -1 Falling back to public interface 2024-06-15T09:48:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:26.889 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:48:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:48:26.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:26.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:27.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:28.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:28.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:28.123 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:48:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:48:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:28.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:28.859 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:48:28.859+0000 7effbddd17c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:48:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:29.364 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:48:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:48:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:29.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:30.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:30.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:30.669 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:48:30.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:48:30.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:30.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:30.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:31.909 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:48:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:48:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:32.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:33.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:33.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:33.149 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:48:33.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:48:33.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:33.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:33.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:34.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:34.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:34.391 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:48:34.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:48:34.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:34.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:34.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:35.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:35.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:35.803 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:48:35.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:48:35.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:35.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:36.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:37.042 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:48:37.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:48:37.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:37.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:37.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:38.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:38.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:38.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:48:38.286 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:48:38.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:38.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:38.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:39.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:39.590 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:48:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:48:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:40.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:41.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:41.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:41.057 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:48:41.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:48:41.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:41.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:41.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:42.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:42.300 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:42.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:43.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:43.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:43.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:48:43.595 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:48:43.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:43.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:43.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:44.853 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:48:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:48:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:45.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:46.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:46.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:46.601 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:48:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:48:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:47.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:48.448 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:48:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:48:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:48.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:49.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:49.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:49.944 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:48:49.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:48:49.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:49.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:50.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:51.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:51.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:51.486 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:48:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:48:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:52.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:52.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:52.962 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:48:52.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:48:52.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:52.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:53.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:54.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:54.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:54.457 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:48:54.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:48:54.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:54.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:54.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:48:55.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:48:55.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:48:55.714 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:48:55.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T09:48:55.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:48:55.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:48:55.953 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3387494331,v1:127.0.0.1:6811/3387494331] [v2:127.0.0.1:6812/3387494331,v1:127.0.0.1:6813/3387494331] exists,up 4805f693-8cf9-4934-ae11-beade780bd92 2024-06-15T09:48:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:48:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:48:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:48:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-06-15T09:48:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T09:48:56.208 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T09:48:56.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T09:48:56.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T09:48:56.684 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T09:48:56.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T09:48:57.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T09:48:59.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-06-15T09:48:59.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:48:59.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:48:59.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:48:59.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:48:59.946 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:48:59.946 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:48:59.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:48:59.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:48:59.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:49:00.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:49:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:49:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T09:49:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:49:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:00.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:49:00.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-06-15T09:49:00.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-06-15T09:49:00.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-06-15T09:49:00.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:00.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:49:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-15T09:49:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-15T09:49:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-06-15T09:49:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:00.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-06-15T09:49:00.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:00.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:49:00.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-06-15T09:49:00.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:00.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-06-15T09:49:00.630 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836492 2024-06-15T09:49:00.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-15T09:49:00.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:49:00.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-06-15T09:49:00.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:49:01.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:49:01.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:49:02.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-15T09:49:02.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:02.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-15T09:49:02.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:02.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:49:02.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-15T09:49:02.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:02.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-15T09:49:02.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-15T09:49:02.105 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672967 2024-06-15T09:49:02.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:49:02.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-15T09:49:02.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:49:02.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:02.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:49:02.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:49:02.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:49:02.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T09:49:02.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:49:02.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:02.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T09:49:03.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T09:49:03.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T09:49:03.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T09:49:03.394 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T09:49:03.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T09:49:03.720 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T09:49:03.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T09:49:03.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T09:49:03.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T09:49:04.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-06-15T09:49:04.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-15T09:49:04.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:49:04.028 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-15T09:49:04.028 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-15T09:49:04.028 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:49:04.029 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-15T09:49:04.029 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:49:04.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:49:04.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-15T09:49:04.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:49:04.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:49:04.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:49:04.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:49:04.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:49:04.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:49:04.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:49:04.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-15T09:49:06.691 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T09:49:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:49:07.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:49:07.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:49:07.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:49:07.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:49:07.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:49:07.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:49:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:49:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:49:07.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:49:07.233 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:49:07.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:49:07.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:49:07.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:49:07.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:49:07.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:49:07.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:49:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:07.270+0000 7fa4280957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:07.274+0000 7fa4280957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:07.274+0000 7fa4280957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:07.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:07.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:49:07.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:07.811 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:07.814+0000 7fa4280957c0 -1 Falling back to public interface 2024-06-15T09:49:08.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:08.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:08.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:49:08.739 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:49:08.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:08.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:49:08.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:49:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:49:09.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:09.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:49:10.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:10.556 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:10.558+0000 7fa4280957c0 -1 osd.0 21 log_to_monitors true 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/801971971,v1:127.0.0.1:6803/801971971] [v2:127.0.0.1:6804/801971971,v1:127.0.0.1:6805/801971971] exists,up 88d4c734-5576-44ce-9cd7-0ad897c2326b 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:49:11.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:49:11.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:49:11.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:49:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:49:11.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:49:11.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:49:11.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:49:11.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:49:11.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:49:11.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:49:11.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:49:11.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:49:11.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:49:12.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:49:12.288 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T09:49:12.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:49:12.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:12.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:49:12.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-06-15T09:49:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-06-15T09:49:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-06-15T09:49:12.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:12.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:49:12.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-15T09:49:12.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-15T09:49:12.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-06-15T09:49:12.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:12.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-06-15T09:49:12.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:12.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:49:12.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-06-15T09:49:12.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:12.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-06-15T09:49:12.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-06-15T09:49:12.642 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 111669149698 2024-06-15T09:49:12.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:49:12.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-06-15T09:49:12.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:49:13.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:49:13.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:49:14.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-06-15T09:49:14.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:14.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-15T09:49:14.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:14.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:49:14.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-15T09:49:14.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:14.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-15T09:49:14.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-15T09:49:14.121 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672970 2024-06-15T09:49:14.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:49:14.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-15T09:49:14.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:49:14.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:14.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:14.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T09:49:14.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:49:14.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:49:14.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:49:14.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:49:14.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:49:14.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:49:14.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:49:14.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T09:49:14.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:49:14.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:14.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T09:49:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:49:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:49:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-15T09:49:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-15T09:49:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T09:49:15.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T09:49:15.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-15T09:49:15.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-15T09:49:15.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-15T09:49:15.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-15T09:49:15.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-15T09:49:15.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:15.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:15.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:15.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:15.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:15.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-15T09:49:15.917 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-15T09:49:15.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:15.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:15.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:15.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:16.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:48:56.555297+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:16.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:49:17.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:49:17.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:17.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:17.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:17.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:17.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:17.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:48:56.555297+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:17.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:49:18.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:49:18.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:18.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:18.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:18.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:18.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:18.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:48:56.555297+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:49:19.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:49:19.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:19.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:48:56.555297+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:19.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:49:20.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:49:20.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:20.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:20.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:20.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:20.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:20.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:21.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:48:56.555297+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:21.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:49:22.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:49:22.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T09:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T09:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:49:22.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T09:49:22.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:49:16.842463+0000 '>' 2024-06-15T09:48:56.555297+0000 2024-06-15T09:49:22.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T09:49:22.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T09:49:22.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:49:22.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:49:22.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T09:49:22.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:49:22.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:49:22.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:49:22.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:49:22.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:49:22.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T09:49:24.123 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T09:49:24.124 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T09:49:24.124 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:49:24.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:49:24.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:49:24.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:49:24.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:49:24.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:49:24.446 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T09:49:24.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:49:24.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T09:49:24.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:49:24.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:49:24.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:49:24.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:49:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:24.482+0000 7fe8e6df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:24.481 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:24.486+0000 7fe8e6df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:24.482 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:24.486+0000 7fe8e6df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:24.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:49:24.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:49:24.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:24.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:49:24.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:25.023 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:25.026+0000 7fe8e6df27c0 -1 Falling back to public interface 2024-06-15T09:49:25.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:49:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:49:25.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:25.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:49:26.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:49:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:49:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:27.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:49:27.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:27.402+0000 7fe8e6df27c0 -1 osd.1 28 log_to_monitors true 2024-06-15T09:49:27.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:49:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:49:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:49:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:49:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:49:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:49:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:49:29.013 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/897549239,v1:127.0.0.1:6811/897549239] [v2:127.0.0.1:6812/897549239,v1:127.0.0.1:6813/897549239] exists,up 4805f693-8cf9-4934-ae11-beade780bd92 2024-06-15T09:49:29.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:49:29.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:49:29.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:49:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:49:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:49:29.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:49:29.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:49:29.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:49:29.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:49:29.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:49:29.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:49:29.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:49:29.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:49:29.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:49:29.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:49:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T09:49:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:49:29.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:29.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:49:29.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-06-15T09:49:29.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-06-15T09:49:29.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-06-15T09:49:29.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:49:29.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-06-15T09:49:29.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:49:29.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-06-15T09:49:29.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-06-15T09:49:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-06-15T09:49:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 111669149702 2024-06-15T09:49:29.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:49:29.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-06-15T09:49:29.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:49:29.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-06-15T09:49:29.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:49:29.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:49:29.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-06-15T09:49:29.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:49:29.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-06-15T09:49:29.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-06-15T09:49:29.932 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 137438953474 2024-06-15T09:49:29.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:49:30.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 137438953474 2024-06-15T09:49:30.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:49:31.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:49:31.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:49:31.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 137438953474 2024-06-15T09:49:31.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:49:32.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T09:49:32.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:49:32.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-06-15T09:49:32.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:49:32.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:32.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T09:49:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:49:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:49:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:49:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:49:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:49:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:49:33.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:49:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T09:49:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:49:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:49:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:49:33.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T09:49:33.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:49:33.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:49:33.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T09:49:33.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T09:49:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T09:49:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:49:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:49:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:49:33.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:49:33.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:49:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:49:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:49:33.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:49:33.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:49:33.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:49:33.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:49:33.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:49:33.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:49:33.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:49:33.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:49:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:49:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:49:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:49:33.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:49:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:49:33.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:49:33.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:49:33.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:49:33.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:49:33.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:49:33.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:49:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:49:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:49:33.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:49:33.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:49:33.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:49:33.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:49:33.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:49:33.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:49:33.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:49:33.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:49:33.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:49:33.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:49:33.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:49:33.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:49:33.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:49:33.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:49:33.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:49:33.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:49:33.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:49:33.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:49:33.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:49:33.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T09:49:33.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:49:33.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:49:33.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:49:33.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T09:49:33.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:49:33.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T09:49:33.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:49:33.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-06-15T09:49:33.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-06-15T09:49:33.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-06-15T09:49:33.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-06-15T09:49:33.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-06-15T09:49:33.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-06-15T09:49:35.704 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 ca144633-2ba7-4eeb-aeca-39d768f41fbb 2024-06-15T09:49:35.964 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:49:36.018 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.022+0000 7fbec240c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:36.019 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.022+0000 7fbec240c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.022+0000 7fbec240c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.026+0000 7fbec240c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:49:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.026+0000 7fbec240c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:49:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.026+0000 7fbec240c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:49:36.115 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:36.118+0000 7fbec240c7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T09:49:42.312 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:49:42.583 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:49:42.615 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:42.618+0000 7fe0733787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:42.618 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:42.622+0000 7fe0733787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:42.619 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:42.622+0000 7fe0733787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:49:42.815 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:49:43.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:43.178+0000 7fe0733787c0 -1 Falling back to public interface 2024-06-15T09:49:44.049 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:49:45.293 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:49:45.887 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:49:45.890+0000 7fe0733787c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:49:46.522 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:49:47.779 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:49:49.019 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:49:50.262 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:49:51.578 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:49:52.902 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:49:54.144 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:49:55.383 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:49:56.814 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:49:58.055 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:49:59.288 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:50:00.570 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:50:01.811 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:50:03.272 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:50:04.988 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:50:06.251 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:50:08.013 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:50:09.404 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:50:11.038 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:50:11.276 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/248501837,v1:127.0.0.1:6803/248501837] [v2:127.0.0.1:6804/248501837,v1:127.0.0.1:6805/248501837] exists,up ca144633-2ba7-4eeb-aeca-39d768f41fbb 2024-06-15T09:50:11.282 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 cc455c70-eb12-4b5f-85e2-ef75a7b2e1a1 2024-06-15T09:50:11.598 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:50:11.654 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.658+0000 7fb1551b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:11.708 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.710+0000 7fb1551b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:11.709 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.714+0000 7fb1551b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:11.710 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.714+0000 7fb1551b67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:50:11.710 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.714+0000 7fb1551b67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:50:11.710 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.714+0000 7fb1551b67c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:50:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:11.798+0000 7fb1551b67c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T09:50:18.761 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:50:19.033 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:50:19.072 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:19.073+0000 7f7cda8f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:19.096 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:19.097+0000 7f7cda8f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:19.097 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:19.101+0000 7f7cda8f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:19.283 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:50:19.652 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:19.653+0000 7f7cda8f37c0 -1 Falling back to public interface 2024-06-15T09:50:20.547 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:50:21.806 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:50:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:22.473+0000 7f7cda8f37c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:50:23.081 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:50:24.364 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:50:25.609 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:50:26.838 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:50:28.158 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:50:29.510 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:50:30.750 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:50:32.062 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:50:33.467 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:50:34.711 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:50:36.053 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:50:37.376 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:50:38.868 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:50:40.469 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:50:42.321 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:50:43.904 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:50:45.734 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:50:47.322 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:50:48.891 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:50:49.172 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3096372130,v1:127.0.0.1:6811/3096372130] [v2:127.0.0.1:6812/3096372130,v1:127.0.0.1:6813/3096372130] exists,up cc455c70-eb12-4b5f-85e2-ef75a7b2e1a1 2024-06-15T09:50:49.178 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 79c81303-48ec-45dd-b68d-4465b8776772 2024-06-15T09:50:49.506 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:50:49.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.565+0000 7f84e78f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:49.608 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.609+0000 7f84e78f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:49.610 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.613+0000 7f84e78f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:49.610 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.613+0000 7f84e78f07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:50:49.610 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.613+0000 7f84e78f07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:50:49.610 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.613+0000 7f84e78f07c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:50:49.685 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:49.685+0000 7f84e78f07c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T09:50:57.116 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:50:57.385 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:50:57.425 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:57.425+0000 7f90f7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:57.457 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:57.457+0000 7f90f7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:57.457 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:57.461+0000 7f90f7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:50:57.647 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:50:58.012 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:50:58.013+0000 7f90f7dae7c0 -1 Falling back to public interface 2024-06-15T09:50:58.888 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:51:00.120 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:51:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:00.705+0000 7f90f7dae7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:51:01.355 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:51:02.718 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:51:04.072 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:51:05.314 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:51:06.644 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:51:08.096 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:51:09.338 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:51:10.690 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:51:12.094 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:51:13.333 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:51:14.687 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:51:16.040 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:51:17.478 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:51:19.094 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:51:20.888 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:51:22.558 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:51:24.208 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:51:25.695 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:51:27.358 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:51:28.601 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:51:28.868 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3661730072,v1:127.0.0.1:6819/3661730072] [v2:127.0.0.1:6820/3661730072,v1:127.0.0.1:6821/3661730072] exists,up 79c81303-48ec-45dd-b68d-4465b8776772 2024-06-15T09:51:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T09:51:31.191 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836505 2024-06-15T09:51:32.670 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672974 2024-06-15T09:51:32.999 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509444 2024-06-15T09:51:38.076 INFO:tasks.workunit.client.0.smithi116.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-06-15T09:51:38.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-06-15T09:51:38.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-06-15T09:51:38.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-06-15T09:51:38.440 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T09:51:38.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-15T09:51:38.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:51:38.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:51:38.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:51:38.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:51:38.463 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:51:38.463 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:51:38.463 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:51:38.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:51:38.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:51:38.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:51:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:51:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:51:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:51:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:51:38.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:38.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:51:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-06-15T09:51:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-06-15T09:51:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-06-15T09:51:39.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:39.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:51:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-06-15T09:51:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-06-15T09:51:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977' 2024-06-15T09:51:39.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:39.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:51:39.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-15T09:51:39.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-15T09:51:39.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509447' 2024-06-15T09:51:39.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:39.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:39.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-06-15T09:51:39.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:51:39.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-06-15T09:51:39.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-06-15T09:51:39.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-06-15T09:51:39.486 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836508 2024-06-15T09:51:39.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:51:39.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-06-15T09:51:39.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:51:40.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:51:40.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:51:40.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-06-15T09:51:40.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:40.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-06-15T09:51:40.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:40.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:51:41.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-06-15T09:51:41.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:41.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-06-15T09:51:41.001 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672977 2024-06-15T09:51:41.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-15T09:51:41.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:51:41.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-06-15T09:51:41.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:41.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-15T09:51:41.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:41.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:51:41.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-15T09:51:41.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:41.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-15T09:51:41.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-15T09:51:41.268 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509447 2024-06-15T09:51:41.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:51:41.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-15T09:51:41.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:51:41.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:51:41.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:51:41.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:51:41.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:51:42.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:51:42.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:51:42.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:51:42.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-06-15T09:51:42.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T09:51:42.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-15T09:51:42.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:51:42.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:51:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:51:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:51:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:51:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:51:42.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:51:42.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:51:42.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:51:42.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:51:43.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:51:43.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:51:43.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:51:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:51:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T09:51:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:51:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:51:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:51:43.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:51:43.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:51:43.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-15T09:51:45.152 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T09:51:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T09:51:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:51:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:51:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:51:45.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:51:45.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:51:45.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:51:45.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:51:45.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:51:45.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:51:45.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:51:45.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:51:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:51:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:51:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:51:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:51:45.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:51:45.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:51:45.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:51:45.728 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:45.728+0000 7fdeafbbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:51:45.731 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:45.732+0000 7fdeafbbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:51:45.732 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:45.732+0000 7fdeafbbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:51:45.970 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:51:45.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:51:45.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:51:46.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:51:46.299 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:46.300+0000 7fdeafbbd7c0 -1 Falling back to public interface 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:51:47.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:51:47.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:51:48.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:51:48.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:51:48.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:51:48.478 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:51:48.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:51:48.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:51:48.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:51:49.025 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:51:49.024+0000 7fdeafbbd7c0 -1 osd.0 21 log_to_monitors true 2024-06-15T09:51:49.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:51:49.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:51:49.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:51:49.722 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:51:49.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:51:49.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:51:49.963 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/932806089,v1:127.0.0.1:6803/932806089] [v2:127.0.0.1:6804/932806089,v1:127.0.0.1:6805/932806089] exists,up ca144633-2ba7-4eeb-aeca-39d768f41fbb 2024-06-15T09:51:49.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:51:49.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:51:49.965 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:51:49.965 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:51:49.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:51:49.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:51:49.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:51:50.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:51:50.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:51:50.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:51:50.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:51:50.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:51:50.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:51:50.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:51:50.304 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:51:50.304 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:51:50.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:51:50.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:50.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:51:50.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-15T09:51:50.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-15T09:51:50.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-15T09:51:50.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:50.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:51:50.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-06-15T09:51:50.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-06-15T09:51:50.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-06-15T09:51:50.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:51:50.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:51:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-06-15T09:51:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-06-15T09:51:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509450' 2024-06-15T09:51:50.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:50.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-15T09:51:50.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:51:50.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-15T09:51:50.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-15T09:51:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-15T09:51:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182402 2024-06-15T09:51:50.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:51:51.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-06-15T09:51:51.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:51:52.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:51:52.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:51:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-15T09:51:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:52.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-06-15T09:51:52.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:52.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:51:52.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-06-15T09:51:52.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-06-15T09:51:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-06-15T09:51:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672980 2024-06-15T09:51:52.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:51:52.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-06-15T09:51:52.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:51:52.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-06-15T09:51:52.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:51:52.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:51:52.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-06-15T09:51:52.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:51:52.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-06-15T09:51:52.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-15T09:51:52.579 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509450 2024-06-15T09:51:52.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:51:52.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-15T09:51:52.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:51:52.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:51:52.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:51:53.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:51:53.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:51:53.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:51:53.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:51:53.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:51:53.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:51:53.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:51:53.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:51:53.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:51:53.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-15T09:51:53.722 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: { 2024-06-15T09:51:53.722 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-15T09:51:53.722 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:51:53.735 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: { 2024-06-15T09:51:53.735 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-15T09:51:53.735 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:51:53.751 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2: { 2024-06-15T09:51:53.751 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-15T09:51:53.751 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:51:53.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-06-15T09:51:53.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T09:51:53.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T09:51:53.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T09:51:53.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-15T09:51:54.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-06-15T09:51:54.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-06-15T09:51:54.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:51:54.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:51:54.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:51:54.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:51:54.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-06-15T09:51:54.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-06-15T09:51:54.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:54.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:54.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:54.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:54.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:54.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-06-15T09:51:54.674 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:51:54.674 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:51:54.675 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T09:51:54.675 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-01T09:50:14.680303+0000" 2024-06-15T09:51:54.675 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T09:51:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:54.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:54.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:54.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:29.226868+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:54.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:51:55.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:51:55.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:51:55.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:51:55.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:55.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:55.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:55.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:56.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:29.226868+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:56.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:51:57.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:51:57.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:51:57.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:51:57.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:57.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:57.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:57.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:57.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:29.226868+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:57.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:51:58.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:51:58.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:51:58.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:51:58.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:58.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:58.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:58.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:58.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:29.226868+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:58.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:51:59.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:51:59.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:51:59.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:51:59.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:51:59.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:51:59.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:51:59.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:51:59.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:29.226868+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:51:59.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:52:00.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:52:00.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:52:00.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:52:00.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:52:00.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:52:00.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:52:00.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:51:54.907039+0000 '>' 2024-06-15T09:51:29.226868+0000 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:52:01.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:52:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:52:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:52:01.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:52:01.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:52:01.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:52:01.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:52:01.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:52:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:52:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:52:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:52:01.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:52:01.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:52:01.452 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:52:01.452 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:52:01.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:52:01.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:01.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:52:01.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-06-15T09:52:01.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-06-15T09:52:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-06-15T09:52:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:01.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:52:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-06-15T09:52:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-06-15T09:52:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672983' 2024-06-15T09:52:01.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:01.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:52:01.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-15T09:52:01.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-15T09:52:01.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672983 2-64424509453' 2024-06-15T09:52:01.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:01.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-06-15T09:52:01.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:01.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:52:01.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-06-15T09:52:01.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:01.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-06-15T09:52:01.929 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 107374182405 2024-06-15T09:52:01.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-06-15T09:52:01.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:52:02.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-06-15T09:52:02.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:02.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-06-15T09:52:02.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:02.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:52:02.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-06-15T09:52:02.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:02.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-06-15T09:52:02.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-06-15T09:52:02.161 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672983 2024-06-15T09:52:02.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:52:02.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-06-15T09:52:02.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:02.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-15T09:52:02.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:02.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:52:02.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-15T09:52:02.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:02.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-15T09:52:02.437 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509453 2024-06-15T09:52:02.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-15T09:52:02.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:02.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509453 2024-06-15T09:52:02.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:52:03.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:52:03.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:03.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509453 2024-06-15T09:52:03.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:52:05.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T09:52:05.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-06-15T09:52:05.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:52:05.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:05.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:05.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:52:05.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:52:05.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:52:05.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:52:05.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:52:05.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:52:05.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:52:05.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:52:05.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:52:05.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:52:05.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:05.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:06.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:52:06.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:52:06.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:52:06.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-06-15T09:52:06.263 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:52:06.263 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T09:51:55.102230+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-06-15T09:51:54.907039+0000 21'1 2024-06-15T09:51:54.907039+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:51:54.907039+0000 1 0 2024-06-15T09:52:06.263 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:52:06.263 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:52:06.264 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:52:06.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-15T09:52:06.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:52:06.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:52:06.281 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:52:06.281 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:52:06.281 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:52:06.282 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:52:06.282 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:52:06.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:52:06.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:52:06.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:52:06.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T09:52:06.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:52:06.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:52:06.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:52:06.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:52:07.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:52:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:52:07.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-15T09:52:08.575 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T09:52:08.575 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T09:52:08.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:52:08.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:52:08.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:52:08.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:52:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:52:08.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:52:08.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:52:08.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:52:08.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:52:08.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:52:08.909 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:08.908+0000 7fdec1ada7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:08.911 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:08.912+0000 7fdec1ada7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:08.913 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:08.912+0000 7fdec1ada7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:52:09.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:09.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:52:09.118 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:52:09.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:09.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:09.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:09.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:09.456+0000 7fdec1ada7c0 -1 Falling back to public interface 2024-06-15T09:52:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:52:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:52:10.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:10.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:10.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:11.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:11.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:11.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:52:11.602 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:52:11.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:11.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:11.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:11.928 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:11.927+0000 7fdec1ada7c0 -1 osd.0 26 log_to_monitors true 2024-06-15T09:52:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:52:12.842 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:52:12.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:12.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:13.124 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1953534772,v1:127.0.0.1:6803/1953534772] [v2:127.0.0.1:6804/1953534772,v1:127.0.0.1:6805/1953534772] exists,up ca144633-2ba7-4eeb-aeca-39d768f41fbb 2024-06-15T09:52:13.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:52:13.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:52:13.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:52:13.126 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:52:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:52:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:52:13.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:52:13.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:52:13.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:52:13.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:52:13.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:52:13.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:52:13.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:13.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:52:13.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T09:52:13.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T09:52:13.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-15T09:52:13.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:13.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:52:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-06-15T09:52:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-06-15T09:52:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986' 2024-06-15T09:52:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:13.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:52:13.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-06-15T09:52:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-06-15T09:52:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986 2-64424509457' 2024-06-15T09:52:13.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:13.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-15T09:52:13.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:13.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:52:13.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-15T09:52:13.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:13.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T09:52:13.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-15T09:52:13.998 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 128849018882 2024-06-15T09:52:13.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:52:14.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T09:52:14.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:14.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:14.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-06-15T09:52:14.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:52:14.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-06-15T09:52:14.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:14.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-06-15T09:52:14.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-15T09:52:14.245 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672986 2024-06-15T09:52:14.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:52:14.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-06-15T09:52:14.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:14.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-06-15T09:52:14.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:14.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:52:14.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-06-15T09:52:14.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-06-15T09:52:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-15T09:52:14.515 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509457 2024-06-15T09:52:14.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:14.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509457 2024-06-15T09:52:14.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:52:15.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:52:15.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:16.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509457 2024-06-15T09:52:16.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:52:17.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T09:52:17.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-15T09:52:17.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:52:17.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:17.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:17.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:52:17.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:52:17.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:52:17.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:52:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:52:17.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:17.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:18.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:52:18.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:52:18.035 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:52:18.036 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:52:18.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:52:18.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:52:18.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:52:18.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:52:18.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:52:18.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:52:18.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:52:18.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:52:18.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:52:18.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T09:52:20.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T09:52:20.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:52:20.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:52:20.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:52:20.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:52:20.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:52:20.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:52:20.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:52:20.556 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:52:20.556 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:20.556 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:20.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:20.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:20.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:52:20.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:52:20.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:52:20.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:52:20.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:52:20.590 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:20.591+0000 7f15e0d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:20.592 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:20.591+0000 7f15e0d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:20.593 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:20.595+0000 7f15e0d047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:52:20.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:20.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:52:20.835 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:52:20.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:20.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:21.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:21.163+0000 7f15e0d047c0 -1 Falling back to public interface 2024-06-15T09:52:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:52:22.095 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:52:22.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:22.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:22.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:23.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:23.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:23.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:52:23.338 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:52:23.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:23.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:23.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:23.705 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:23.707+0000 7f15e0d047c0 -1 osd.0 31 log_to_monitors true 2024-06-15T09:52:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:52:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:52:24.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:24.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/3809493665,v1:127.0.0.1:6803/3809493665] [v2:127.0.0.1:6804/3809493665,v1:127.0.0.1:6805/3809493665] exists,up ca144633-2ba7-4eeb-aeca-39d768f41fbb 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:52:24.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:52:24.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:52:24.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:52:24.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:52:24.854 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:52:24.854 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:52:24.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:52:24.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:52:24.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:52:24.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:52:24.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:52:24.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:52:24.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:52:24.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:52:24.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:52:25.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:52:25.225 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:52:25.225 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:52:25.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:52:25.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:25.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:52:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-15T09:52:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-15T09:52:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-06-15T09:52:25.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:25.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:52:25.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-06-15T09:52:25.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-06-15T09:52:25.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990' 2024-06-15T09:52:25.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:52:25.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990 2-64424509460' 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-06-15T09:52:25.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:25.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:52:25.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-06-15T09:52:25.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:25.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-15T09:52:25.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-06-15T09:52:25.751 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 150323855362 2024-06-15T09:52:25.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:52:25.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018883 -lt 150323855362 2024-06-15T09:52:25.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:52:26.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:52:26.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:52:27.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-15T09:52:27.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:27.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-06-15T09:52:27.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:27.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:52:27.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-06-15T09:52:27.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:27.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-06-15T09:52:27.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-06-15T09:52:27.228 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672990 2024-06-15T09:52:27.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:52:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-06-15T09:52:27.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:52:27.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:52:27.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-06-15T09:52:27.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:52:27.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-06-15T09:52:27.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:52:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-06-15T09:52:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-15T09:52:27.500 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509460 2024-06-15T09:52:27.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:52:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-15T09:52:27.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:52:27.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:27.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:28.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:52:28.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:52:28.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:52:28.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:52:28.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:52:28.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:52:28.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:52:28.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:52:28.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:52:28.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:52:28.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T09:52:28.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T09:52:28.545 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:51:54.900+0000 7f7cbe87d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:52:28.545 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:51:54.900+0000 7f7cbe87d640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T09:52:28.545 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:51:54.900+0000 7f7cbe87d640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-15T09:52:28.546 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:51:54.900+0000 7f7cbe87d640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-15T09:52:28.546 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:51:54.900+0000 7f7cbe87d640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:52:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:52:28.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:52:28.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:52:28.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:52:28.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:52:28.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:52:28.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:52:28.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:52:28.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:52:28.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:52:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:52:28.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:52:28.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:52:28.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:52:28.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:52:28.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:52:28.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:52:28.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:28.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:28.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:52:28.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:52:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:52:28.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:52:28.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:52:28.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:52:28.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:52:28.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:52:28.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:52:28.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:52:28.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:52:28.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:52:28.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:52:28.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:52:28.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:52:28.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:52:28.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:52:28.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:52:28.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:52:28.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:52:28.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:52:28.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:52:28.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:28.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:28.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:52:28.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:52:28.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:52:28.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T09:52:28.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:52:28.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:28.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:28.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-06-15T09:52:28.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T09:52:28.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T09:52:29.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:52:29.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:52:29.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:52:29.294 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:52:29.294 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:29.294 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:29.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:29.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T09:52:29.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:52:29.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:52:29.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:52:29.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:52:29.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:52:29.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:52:29.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:52:29.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:52:29.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:52:29.516 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:52:29.516 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:29.516 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:29.516 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:52:29.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:52:29.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T09:52:29.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:52:29.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:52:29.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:52:29.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:52:29.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:52:29.636 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:52:29.636 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:52:29.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:52:29.637 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:52:29.637 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:29.637 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:29.638 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:52:29.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:52:29.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T09:52:29.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:30.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:52:30.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:52:30.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:52:30.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:52:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:52:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:52:30.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:52:30.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:52:30.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:52:30.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 150920bc-f946-4c2a-8c84-708ab797a0b9' 2024-06-15T09:52:30.664 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:52:30.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:52:30.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeZG1mS5LUKBAAwmbRDmf7nsRU7rUHT31OIw== 2024-06-15T09:52:30.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeZG1mS5LUKBAAwmbRDmf7nsRU7rUHT31OIw=="}' 2024-06-15T09:52:30.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 150920bc-f946-4c2a-8c84-708ab797a0b9 -i td/osd-scrub-repair/0/new.json 2024-06-15T09:52:30.957 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:52:30.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T09:52:30.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBeZG1mS5LUKBAAwmbRDmf7nsRU7rUHT31OIw== --osd-uuid 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:52:31.007 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.007+0000 7f8916e6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:31.024 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.023+0000 7f8916e6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:31.025 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.023+0000 7f8916e6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:31.026 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.027+0000 7f8916e6f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:52:31.026 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.027+0000 7f8916e6f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:52:31.026 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.027+0000 7f8916e6f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:52:31.121 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:31.119+0000 7f8916e6f7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T09:52:37.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T09:52:37.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:52:37.646 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:52:37.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:52:37.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:52:37.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:52:37.917 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:52:37.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:52:37.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:52:37.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:52:37.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:52:37.950 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:37.951+0000 7ff8ee8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:37.952 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:37.951+0000 7ff8ee8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:37.953 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:37.955+0000 7ff8ee8ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:38.178 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:52:38.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:52:38.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:38.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:38.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:38.511 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:38.511+0000 7ff8ee8ec7c0 -1 Falling back to public interface 2024-06-15T09:52:39.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:39.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:39.465 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:52:39.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:52:39.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:39.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:39.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:40.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:40.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:40.701 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:52:40.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:52:40.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:40.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:40.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:41.220 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:52:41.219+0000 7ff8ee8ec7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:52:41.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:41.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:41.945 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:52:41.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:52:41.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:41.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:42.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:43.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:43.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:43.332 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:52:43.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:52:43.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:43.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:43.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:44.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:44.572 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:52:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:52:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:44.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:44.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:45.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:45.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:45.859 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:52:45.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:52:45.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:45.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:47.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:47.203 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:52:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:52:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:47.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:47.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:48.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:48.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:48.444 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:52:48.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:52:48.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:48.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:48.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:49.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:49.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:49.687 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:52:49.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:52:49.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:49.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:50.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:51.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:51.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:51.080 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:52:51.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:52:51.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:51.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:51.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:52.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:52.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:52.324 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:52:52.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:52:52.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:52.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:52.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:53.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:53.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:53.567 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:52:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:52:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:53.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:54.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:54.863 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:52:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:52:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:55.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:56.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:56.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:56.677 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:52:56.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:52:56.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:56.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:57.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:57.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:57.923 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:52:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:52:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:52:58.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:52:59.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:52:59.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:52:59.442 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:52:59.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:52:59.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:52:59.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:00.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:01.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:01.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:01.562 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:53:01.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:53:01.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:01.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:02.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:03.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:03.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:03.231 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:53:03.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:53:03.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:03.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:03.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:04.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:04.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:04.475 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:53:04.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:53:04.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:04.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:04.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:06.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:06.001 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:53:06.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:53:06.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:06.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:06.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:07.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:07.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:07.283 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:53:07.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:53:07.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:07.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:53:07.532 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/827525672,v1:127.0.0.1:6803/827525672] [v2:127.0.0.1:6804/827525672,v1:127.0.0.1:6805/827525672] exists,up 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:53:07.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:53:07.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:53:07.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:53:07.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:53:07.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:53:07.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:53:07.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:53:07.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d95d4847-728c-4bd0-87be-7ba253858a4c' 2024-06-15T09:53:07.542 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:53:07.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:53:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDZG1m90OsIRAA2eY/paOyj2lgXCmcKkRprQ== 2024-06-15T09:53:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDZG1m90OsIRAA2eY/paOyj2lgXCmcKkRprQ=="}' 2024-06-15T09:53:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d95d4847-728c-4bd0-87be-7ba253858a4c -i td/osd-scrub-repair/1/new.json 2024-06-15T09:53:07.853 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:53:07.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T09:53:07.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCDZG1m90OsIRAA2eY/paOyj2lgXCmcKkRprQ== --osd-uuid d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:53:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.907+0000 7feda603e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:07.944 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.947+0000 7feda603e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:07.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.951+0000 7feda603e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:07.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.951+0000 7feda603e7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:53:07.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.951+0000 7feda603e7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:53:07.946 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:07.951+0000 7feda603e7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:53:08.036 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:08.039+0000 7feda603e7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T09:53:14.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T09:53:14.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:53:14.845 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:53:14.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:53:14.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:53:15.120 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:53:15.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:53:15.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:53:15.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:53:15.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:53:15.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:53:15.154 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:15.159+0000 7f1d568567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:15.190 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:15.195+0000 7f1d568567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:15.192 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:15.195+0000 7f1d568567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:15.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:53:15.387 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:53:15.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:15.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:15.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:15.752 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:15.755+0000 7f1d568567c0 -1 Falling back to public interface 2024-06-15T09:53:16.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:16.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:16.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:53:16.649 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:53:16.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:16.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:16.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:17.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:17.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:17.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:53:17.884 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:53:17.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:17.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:18.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:18.452 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:18.454+0000 7f1d568567c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:53:19.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:19.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:19.127 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:53:19.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:53:19.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:19.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:19.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:20.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:20.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:20.408 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:53:20.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:53:20.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:20.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:20.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:21.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:21.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:21.745 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:53:21.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:53:21.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:21.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:22.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:23.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:23.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:23.056 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:53:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:53:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:23.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:23.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:24.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:24.411 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:53:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:53:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:24.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:24.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:25.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:25.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:25.653 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:53:25.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:53:25.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:25.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:26.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:27.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:27.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:27.040 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:53:27.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:53:27.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:27.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:27.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:28.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:28.401 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:53:28.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:53:28.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:28.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:28.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:29.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:29.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:29.642 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:53:29.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:53:29.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:29.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:30.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:31.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:31.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:31.049 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:53:31.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:53:31.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:31.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:31.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:32.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:32.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:32.436 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:53:32.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:53:32.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:32.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:32.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:33.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:33.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:33.682 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:53:33.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:53:33.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:33.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:34.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:35.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:35.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:35.075 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:53:35.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:53:35.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:35.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:35.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:36.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:36.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:36.511 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:53:36.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:53:36.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:36.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:37.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:38.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:38.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:38.259 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:53:38.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:53:38.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:38.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:38.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:39.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:39.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:39.878 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:53:39.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:53:39.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:39.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:40.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:41.239 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:53:41.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:53:41.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:41.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:41.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:42.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:42.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:42.645 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:53:42.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:53:42.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:42.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:43.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:44.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:44.356 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:53:44.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:53:44.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:44.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:44.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:45.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:45.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:45.609 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:53:45.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T09:53:45.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:45.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:53:45.878 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3917305905,v1:127.0.0.1:6811/3917305905] [v2:127.0.0.1:6812/3917305905,v1:127.0.0.1:6813/3917305905] exists,up d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:53:45.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:53:45.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:53:45.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:53:45.882 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:53:45.882 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:53:45.882 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:53:45.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:53:45.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:53:45.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:53:45.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T09:53:45.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:53:45.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a3b8551-9322-479c-a919-ac36a39b8d65 2024-06-15T09:53:45.885 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 0a3b8551-9322-479c-a919-ac36a39b8d65 2024-06-15T09:53:45.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0a3b8551-9322-479c-a919-ac36a39b8d65' 2024-06-15T09:53:45.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:53:45.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpZG1m15caNhAAhJ8Q7sXntudMtAXA6ynsVQ== 2024-06-15T09:53:45.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpZG1m15caNhAAhJ8Q7sXntudMtAXA6ynsVQ=="}' 2024-06-15T09:53:45.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a3b8551-9322-479c-a919-ac36a39b8d65 -i td/osd-scrub-repair/2/new.json 2024-06-15T09:53:46.207 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:53:46.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T09:53:46.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCpZG1m15caNhAAhJ8Q7sXntudMtAXA6ynsVQ== --osd-uuid 0a3b8551-9322-479c-a919-ac36a39b8d65 2024-06-15T09:53:46.255 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.258+0000 7f8c3cce47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:46.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.318+0000 7f8c3cce47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:46.319 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.322+0000 7f8c3cce47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:46.319 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.322+0000 7f8c3cce47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:53:46.319 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.322+0000 7f8c3cce47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:53:46.319 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.322+0000 7f8c3cce47c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:53:46.405 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:46.406+0000 7f8c3cce47c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T09:53:53.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T09:53:53.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:53:53.321 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:53:53.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T09:53:53.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:53:53.588 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:53:53.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T09:53:53.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:53:53.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:53:53.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:53:53.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:53:53.622 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:53.626+0000 7f0d044d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:53.656 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:53.658+0000 7f0d044d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:53.658 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:53.662+0000 7f0d044d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:53.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:53:53.854 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:53:53.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:53.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:53:54.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:54.211 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:54.214+0000 7f0d044d87c0 -1 Falling back to public interface 2024-06-15T09:53:55.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:55.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:55.109 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:53:55.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:53:55.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:55.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:53:55.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:56.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:56.351 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:53:56.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:56.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:53:56.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:56.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:53:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:57.013 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:53:57.014+0000 7f0d044d87c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:53:57.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:57.595 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:53:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:53:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:53:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:53:58.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:53:58.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:53:58.942 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:53:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:53:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:53:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:53:59.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:00.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:00.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:00.187 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:54:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:54:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:00.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:01.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:01.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:01.436 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:54:01.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:54:01.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:01.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:01.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:02.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:02.731 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:02.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:03.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:04.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:04.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:04.077 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:54:04.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:54:04.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:04.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:04.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:05.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:05.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:05.428 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:54:05.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:54:05.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:05.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:05.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:06.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:06.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:06.746 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:54:06.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:54:06.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:06.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:07.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:08.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:08.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:08.040 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:54:08.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:54:08.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:08.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:08.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:09.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:09.426 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:54:09.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:54:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:09.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:10.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:10.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:10.709 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:54:10.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:54:10.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:10.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:10.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:11.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:11.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:11.954 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:54:11.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:54:11.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:11.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:12.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:13.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:13.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:13.506 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:54:13.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:54:13.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:13.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:14.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:15.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:15.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:15.103 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:15.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:15.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:16.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:16.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:16.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:54:16.632 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:54:16.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:16.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:17.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:18.344 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:54:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:54:18.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:18.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:18.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:19.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:19.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:19.998 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:54:19.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:54:19.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:19.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:21.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:21.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:21.737 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:54:21.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:54:21.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:21.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:23.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:23.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:23.471 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:54:23.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:54:23.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:23.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:23.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:24.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:24.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:24.752 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:54:24.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T09:54:24.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:24.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:54:25.009 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1600918072,v1:127.0.0.1:6819/1600918072] [v2:127.0.0.1:6820/1600918072,v1:127.0.0.1:6821/1600918072] exists,up 0a3b8551-9322-479c-a919-ac36a39b8d65 2024-06-15T09:54:25.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:54:25.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:54:25.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:54:25.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-06-15T09:54:25.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-15T09:54:25.362 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T09:54:25.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T09:54:26.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-06-15T09:54:26.732 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T09:54:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-15T09:54:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:54:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:54:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:54:26.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:54:26.754 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:54:26.754 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:54:26.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:54:26.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:54:26.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:54:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:54:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:54:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:54:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:54:26.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:54:26.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:27.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:54:27.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-06-15T09:54:27.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-06-15T09:54:27.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-06-15T09:54:27.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:27.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:54:27.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-15T09:54:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-15T09:54:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672974' 2024-06-15T09:54:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:27.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:54:27.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-15T09:54:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-15T09:54:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672974 2-64424509448' 2024-06-15T09:54:27.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:27.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-06-15T09:54:27.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:27.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:54:27.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-06-15T09:54:27.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:27.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-06-15T09:54:27.745 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836501 2024-06-15T09:54:27.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-15T09:54:27.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:54:28.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-06-15T09:54:28.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:54:29.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:54:29.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:54:29.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-15T09:54:29.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:29.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-15T09:54:29.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:29.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:54:29.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-15T09:54:29.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:29.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-15T09:54:29.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-15T09:54:29.269 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672974 2024-06-15T09:54:29.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:54:29.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-15T09:54:29.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:29.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-15T09:54:29.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:29.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:54:29.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-15T09:54:29.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:29.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-15T09:54:29.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-15T09:54:29.520 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509448 2024-06-15T09:54:29.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:54:29.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-15T09:54:29.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:54:29.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:29.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:30.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:54:30.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:30.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:30.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:54:30.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:30.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:30.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:30.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:54:30.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T09:54:30.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T09:54:30.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T09:54:30.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:54:30.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T09:54:30.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:54:30.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:30.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:30.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:30.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:54:30.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:30.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:30.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:54:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:54:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:54:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-06-15T09:54:31.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-06-15T09:54:31.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-06-15T09:54:31.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:31.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:31.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:31.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:31.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:31.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:32.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:32.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:32.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:32.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:32.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:32.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:32.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-06-15T09:54:32.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:54:32.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-15T09:54:32.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:54:32.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:54:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:54:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:54:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:54:32.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:54:32.769 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:54:33.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:54:33.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:54:33.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:54:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:54:33.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:54:33.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:54:33.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-15T09:54:35.078 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-15T09:54:35.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:54:35.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:54:35.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:54:35.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:54:35.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:54:35.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:54:35.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:54:35.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:54:35.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:54:35.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:54:35.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:54:35.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:54:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:54:35.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:54:35.656 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:35.657+0000 7fcafd0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:35.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:35.661+0000 7fcafd0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:35.660 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:35.661+0000 7fcafd0537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:54:35.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:35.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:54:35.887 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:54:35.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:35.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:36.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:36.196 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:36.197+0000 7fcafd0537c0 -1 Falling back to public interface 2024-06-15T09:54:37.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:37.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:37.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:54:37.139 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:54:37.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:37.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:37.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:38.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:38.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:38.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:54:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:54:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:38.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:38.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:38.862 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:38.865+0000 7fcafd0537c0 -1 osd.0 21 log_to_monitors true 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:39.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2288958227,v1:127.0.0.1:6803/2288958227] [v2:127.0.0.1:6804/2288958227,v1:127.0.0.1:6805/2288958227] exists,up 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:54:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:54:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:54:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:54:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:54:39.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:54:39.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:54:39.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:54:40.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:40.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:54:40.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-15T09:54:40.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-15T09:54:40.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-15T09:54:40.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:40.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:54:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-06-15T09:54:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-06-15T09:54:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-06-15T09:54:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:40.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:54:40.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-06-15T09:54:40.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-06-15T09:54:40.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509452' 2024-06-15T09:54:40.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:40.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-15T09:54:40.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:40.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:54:40.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-15T09:54:40.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:40.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-15T09:54:40.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-15T09:54:40.934 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182402 2024-06-15T09:54:40.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:54:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-15T09:54:41.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:41.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-06-15T09:54:41.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:41.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:54:41.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-06-15T09:54:41.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-06-15T09:54:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-15T09:54:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672978 2024-06-15T09:54:41.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:54:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-15T09:54:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:41.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-06-15T09:54:41.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:41.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:54:41.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-06-15T09:54:41.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:41.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-06-15T09:54:41.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-06-15T09:54:41.433 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509452 2024-06-15T09:54:41.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:54:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509452 2024-06-15T09:54:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:54:42.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:54:42.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:54:43.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-06-15T09:54:43.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:54:43.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:43.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:43.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:54:43.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:43.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:43.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:43.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:43.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:43.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:43.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:43.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:54:43.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:43.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:43.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:54:43.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:54:43.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:54:43.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:54:44.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:54:44.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:54:44.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:54:44.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:54:44.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:54:44.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:54:44.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:54:44.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:54:44.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:54:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:54:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:54:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:54:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:54:44.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:54:44.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:54:44.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:54:44.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-15T09:54:46.157 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:104778fc:::obj2:head# 2024-06-15T09:54:46.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:54:46.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:54:46.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:54:46.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:54:46.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:54:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:54:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:54:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:54:46.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:54:46.743 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:46.745+0000 7f4dfddb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:46.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:46.749+0000 7f4dfddb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:46.747 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:46.749+0000 7f4dfddb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:54:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:54:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:47.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:47.296 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:47.297+0000 7f4dfddb67c0 -1 Falling back to public interface 2024-06-15T09:54:48.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:48.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:48.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:54:48.178 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:54:48.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:48.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:48.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:49.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:49.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:54:49.971 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:54:49.973+0000 7f4dfddb67c0 -1 osd.0 26 log_to_monitors true 2024-06-15T09:54:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:54:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:54:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:54:50.660 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:54:50.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:54:50.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3658913409,v1:127.0.0.1:6803/3658913409] [v2:127.0.0.1:6804/3658913409,v1:127.0.0.1:6805/3658913409] exists,up 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:54:50.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:54:50.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:54:50.907 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:54:50.908 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:54:50.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:54:50.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:54:50.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:54:51.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:54:51.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:54:51.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:54:51.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:54:51.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:54:51.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:51.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:54:51.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T09:54:51.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T09:54:51.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-15T09:54:51.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:51.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:54:51.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-15T09:54:51.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-15T09:54:51.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981' 2024-06-15T09:54:51.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:54:51.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:54:51.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-06-15T09:54:51.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-06-15T09:54:51.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981 2-64424509455' 2024-06-15T09:54:51.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:51.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-15T09:54:51.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:51.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:54:51.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-15T09:54:51.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:51.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T09:54:51.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-15T09:54:51.749 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 128849018882 2024-06-15T09:54:51.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:54:52.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-06-15T09:54:52.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:54:53.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:54:53.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:54:53.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T09:54:53.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:53.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-15T09:54:53.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:53.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:54:53.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-15T09:54:53.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:53.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-15T09:54:53.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-15T09:54:53.251 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672981 2024-06-15T09:54:53.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:54:53.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-15T09:54:53.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:54:53.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-06-15T09:54:53.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:54:53.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:54:53.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-06-15T09:54:53.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:54:53.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-06-15T09:54:53.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-15T09:54:53.524 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509455 2024-06-15T09:54:53.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:54:53.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-15T09:54:53.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:54:53.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:53.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:54.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:54:54.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:54.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:54.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:54.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:54.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:54.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:54.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:54.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:54:54.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:54.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:54.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:54.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:54:54.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T09:54:54.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T09:54:54.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T09:54:54.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:54:54.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T09:54:54.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:54:54.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:54.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:54.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:54.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:54.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:54.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:54.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:54.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:54:54.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:54.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:54.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T09:54:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T09:54:55.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T09:54:55.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T09:54:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T09:54:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T09:54:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T09:54:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:54:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T09:54:55.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:54:55.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:55.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:55.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:54:55.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:55.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:55.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:56.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T09:54:56.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T09:54:56.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T09:54:56.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T09:54:56.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T09:54:56.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T09:54:56.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:54:56.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T09:54:56.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:54:56.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:54:56.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:54:56.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:54:56.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:54:56.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:54:56.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:54:56.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:54:57.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:54:57.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:54:57.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:54:57.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:54:57.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:54:57.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:54:57.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:54:57.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-15T09:54:57.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:54:57.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:54:57.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:54:57.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:54:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:54:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:54:58.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:55:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:55:00.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:55:00.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:55:00.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:00.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:55:00.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:55:00.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:55:00.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:55:00.828 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T09:55:00.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:00.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T09:55:00.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:55:00.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:55:00.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:55:00.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:55:00.865 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:00.869+0000 7f87d35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:00.895 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:00.897+0000 7f87d35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:00.896 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:00.897+0000 7f87d35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:55:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:55:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:55:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:55:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:55:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:55:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:55:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:01.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:01.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:01.467 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:01.469+0000 7f87d35117c0 -1 Falling back to public interface 2024-06-15T09:55:02.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:55:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:02.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:02.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:03.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:03.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:03.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:55:03.613 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:55:03.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:03.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:03.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:04.252 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:04.253+0000 7f87d35117c0 -1 osd.1 31 log_to_monitors true 2024-06-15T09:55:04.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:04.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:04.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:55:04.991 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:55:04.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:04.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:05.469 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/3218870903,v1:127.0.0.1:6811/3218870903] [v2:127.0.0.1:6812/3218870903,v1:127.0.0.1:6813/3218870903] exists,up d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:55:05.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:55:05.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:55:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:55:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:55:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:55:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:55:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:55:05.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:55:05.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:55:05.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:55:05.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:55:05.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:55:05.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:05.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:05.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:05.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:06.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018886 2024-06-15T09:55:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018886 2024-06-15T09:55:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886' 2024-06-15T09:55:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:06.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:06.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-15T09:55:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-15T09:55:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886 1-150323855362' 2024-06-15T09:55:06.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:06.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:06.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-15T09:55:06.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-15T09:55:06.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886 1-150323855362 2-64424509459' 2024-06-15T09:55:06.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:06.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018886 2024-06-15T09:55:06.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:06.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018886 2024-06-15T09:55:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:06.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018886 2024-06-15T09:55:06.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018886' 2024-06-15T09:55:06.305 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 128849018886 2024-06-15T09:55:06.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:06.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018886 2024-06-15T09:55:06.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:55:07.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:07.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018886 2024-06-15T09:55:07.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:07.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-06-15T09:55:07.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:07.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:07.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-06-15T09:55:07.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:07.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-15T09:55:07.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-15T09:55:07.788 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 150323855362 2024-06-15T09:55:07.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-15T09:55:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:08.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-15T09:55:08.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:08.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:08.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-15T09:55:08.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:08.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-15T09:55:08.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-15T09:55:08.061 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509459 2024-06-15T09:55:08.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:08.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-15T09:55:08.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:55:08.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:08.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:08.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:55:08.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:55:08.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:55:08.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:55:08.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:55:08.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:55:08.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:08.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:09.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:55:09.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-15T09:55:09.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T09:55:09.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-06-15T09:55:09.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-06-15T09:55:09.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:55:09.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T09:55:09.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T09:55:09.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:55:09.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-06-15T09:55:09.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-06-15T09:55:09.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:09.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:09.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:09.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:09.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:09.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-06-15T09:55:10.035 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:55:10.035 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:55:10.035 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T09:55:10.035 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-01T09:53:30.040971+0000" 2024-06-15T09:55:10.035 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:55:10.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:10.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T09:55:10.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:10.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T09:55:10.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T09:55:10.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:10.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:10.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:10.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:10.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:10.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:10.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:54:25.297775+0000 '>' 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:10.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:11.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:11.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:11.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:11.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:11.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:11.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:11.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:11.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:54:25.297775+0000 '>' 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:11.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:12.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:12.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:12.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:12.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:12.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:12.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:12.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:12.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:54:25.297775+0000 '>' 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:12.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:13.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:13.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:13.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:13.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:13.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:13.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:13.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:14.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:54:25.297775+0000 '>' 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:14.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:15.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:15.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:15.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:15.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:15.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:15.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:15.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:15.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:54:25.297775+0000 2024-06-15T09:55:15.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T09:55:15.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-06-15T09:55:15.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:55:15.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:55:15.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:55:15.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:55:15.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:55:15.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:55:15.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:55:15.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:55:15.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:15.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:15.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:15.603 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:15.603 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:15.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:15.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:15.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:15.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018889 2024-06-15T09:55:15.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018889 2024-06-15T09:55:15.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889' 2024-06-15T09:55:15.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:15.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:15.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-06-15T09:55:15.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-06-15T09:55:15.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365' 2024-06-15T09:55:15.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:15.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:16.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-06-15T09:55:16.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-06-15T09:55:16.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855365 2-64424509462' 2024-06-15T09:55:16.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:16.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018889 2024-06-15T09:55:16.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:16.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018889 2024-06-15T09:55:16.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:16.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018889 2024-06-15T09:55:16.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018889' 2024-06-15T09:55:16.093 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 128849018889 2024-06-15T09:55:16.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:16.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018889 2024-06-15T09:55:16.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:17.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:55:17.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:17.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018889 2024-06-15T09:55:17.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:17.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-06-15T09:55:17.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:17.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:17.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-06-15T09:55:17.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:17.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-06-15T09:55:17.571 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 150323855365 2024-06-15T09:55:17.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-06-15T09:55:17.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:17.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-15T09:55:17.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:17.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-06-15T09:55:17.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:17.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:17.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-06-15T09:55:17.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:17.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-06-15T09:55:17.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-15T09:55:17.832 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509462 2024-06-15T09:55:17.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:18.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-06-15T09:55:18.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:55:18.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:18.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:55:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:55:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:55:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:55:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:55:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:55:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:55:18.349 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:55:18.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:55:18.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:55:18.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:18.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-15T09:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:19.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:19.175 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:19.175 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:19.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:19.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:19.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:19.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-06-15T09:55:19.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-06-15T09:55:19.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-06-15T09:55:19.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:19.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:19.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855366 2024-06-15T09:55:19.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855366 2024-06-15T09:55:19.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855366' 2024-06-15T09:55:19.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:19.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:19.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-15T09:55:19.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-15T09:55:19.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855366 2-64424509464' 2024-06-15T09:55:19.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:19.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-06-15T09:55:19.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:19.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:19.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-06-15T09:55:19.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:19.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-06-15T09:55:19.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-06-15T09:55:19.602 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 128849018890 2024-06-15T09:55:19.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:19.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-06-15T09:55:19.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:20.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:55:20.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:21.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018891 -lt 128849018890 2024-06-15T09:55:21.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855366 2024-06-15T09:55:21.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:21.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:21.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855366 2024-06-15T09:55:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:21.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855366 2024-06-15T09:55:21.061 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 150323855366 2024-06-15T09:55:21.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855366' 2024-06-15T09:55:21.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:21.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855366 2024-06-15T09:55:21.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:21.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-15T09:55:21.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:21.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:21.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:21.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-15T09:55:21.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-15T09:55:21.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-15T09:55:21.328 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509464 2024-06-15T09:55:21.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:21.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-15T09:55:21.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T09:55:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T09:55:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-15T09:55:21.575 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-15T09:55:21.576 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:21.576 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:21.576 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-15T09:55:21.576 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:21.576 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-15T09:55:21.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-06-15T09:55:21.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-15T09:55:21.819 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:55:21.819 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-15T09:55:10.702246+0000 21'10 36:96 [1,0] 1 [1,0] 1 21'10 2024-06-15T09:55:10.701860+0000 21'10 2024-06-15T09:55:10.701860+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:55:10.701860+0000 10 0 2024-06-15T09:55:21.820 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:55:21.820 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:55:21.820 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:55:21.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-15T09:55:21.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-06-15T09:55:22.038 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:55:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-06-15T09:55:22.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:55:22.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-15T09:55:22.054 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-15T09:55:22.054 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:55:22.054 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T09:55:22.055 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T09:55:22.055 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:55:22.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:55:22.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-15T09:55:22.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:55:22.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-06-15T09:55:22.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:55:22.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:55:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:55:22.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:55:23.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:55:23.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-06-15T09:55:24.570 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T09:55:24.570 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T09:55:25.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:25.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:55:25.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:55:25.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:55:25.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:55:25.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:55:25.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:25.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:25.111 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:25.111 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:55:25.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:55:25.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:55:25.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:55:25.113 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:55:25.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:25.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:55:25.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:55:25.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:55:25.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:55:25.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:55:25.147 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:25.149+0000 7f94bcef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:25.150 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:25.153+0000 7f94bcef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:25.152 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:25.153+0000 7f94bcef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:25.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:25.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:55:25.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:25.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:25.701+0000 7f94bcef87c0 -1 Falling back to public interface 2024-06-15T09:55:26.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:26.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:26.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:55:26.612 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:26.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:26.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:55:26.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:27.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:27.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:27.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:55:27.854 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:55:27.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:27.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:55:28.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:28.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:28.425+0000 7f94bcef87c0 -1 osd.0 36 log_to_monitors true 2024-06-15T09:55:29.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:29.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:29.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:55:29.092 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:55:29.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:29.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/2586266960,v1:127.0.0.1:6803/2586266960] [v2:127.0.0.1:6804/2586266960,v1:127.0.0.1:6805/2586266960] exists,up 150920bc-f946-4c2a-8c84-708ab797a0b9 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:55:29.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:55:29.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:55:29.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:55:29.379 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:55:29.379 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:55:29.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:55:29.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:55:29.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:29.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:29.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:29.753 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:29.753 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:29.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:29.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:29.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:29.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T09:55:29.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T09:55:29.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-06-15T09:55:29.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:29.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:30.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-06-15T09:55:30.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-06-15T09:55:30.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-06-15T09:55:30.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:30.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:30.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509467' 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:30.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:30.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-06-15T09:55:30.280 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:30.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T09:55:30.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-06-15T09:55:30.281 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 171798691842 2024-06-15T09:55:30.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:30.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-06-15T09:55:30.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:31.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:55:31.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:31.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-15T09:55:31.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:31.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-06-15T09:55:31.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:31.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:31.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-06-15T09:55:31.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:31.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-06-15T09:55:31.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-06-15T09:55:31.756 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 150323855370 2024-06-15T09:55:31.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:31.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-06-15T09:55:31.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:31.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-06-15T09:55:31.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:31.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:31.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:31.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-06-15T09:55:31.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-06-15T09:55:31.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-06-15T09:55:31.992 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509467 2024-06-15T09:55:31.992 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:32.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-06-15T09:55:32.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:55:32.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:32.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:32.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:55:32.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:55:32.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:55:32.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:55:32.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:55:32.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:32.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:55:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:55:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:55:33.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-06-15T09:55:33.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T09:55:33.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:33.112 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-15T09:55:33.113 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:33.113 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-15T09:55:33.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-06-15T09:55:33.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:55:33.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:55:33.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:55:33.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:55:33.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-06-15T09:55:33.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:55:33.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:55:33.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:55:33.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:55:33.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:55:33.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:55:33.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-06-15T09:55:33.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:55:33.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:55:33.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T09:55:33.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:55:33.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:55:33.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-06-15T09:55:35.327 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-06-15T09:55:35.327 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:104778fc:::obj2:head# 2024-06-15T09:55:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:55:35.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:55:35.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:35.865 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:35.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:35.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T09:55:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:35.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T09:55:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:55:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:55:35.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:55:35.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:55:35.902 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:35.905+0000 7f8ad86ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:35.905 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:35.905+0000 7f8ad86ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:35.906 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:35.909+0000 7f8ad86ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:55:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:55:36.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:36.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:36.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:36.439 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:36.441+0000 7f8ad86ae7c0 -1 Falling back to public interface 2024-06-15T09:55:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:55:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:37.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:37.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:37.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:38.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:55:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:55:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:38.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:55:39.166 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:39.169+0000 7f8ad86ae7c0 -1 osd.1 41 log_to_monitors true 2024-06-15T09:55:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:55:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:55:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:55:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:55:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:55:39.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/2240422722,v1:127.0.0.1:6811/2240422722] [v2:127.0.0.1:6812/2240422722,v1:127.0.0.1:6813/2240422722] exists,up d95d4847-728c-4bd0-87be-7ba253858a4c 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:55:40.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:55:40.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:55:40.170 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:55:40.170 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:55:40.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:55:40.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:55:40.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:40.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:40.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:40.606 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:40.606 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:40.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:40.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:40.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:40.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-06-15T09:55:40.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-06-15T09:55:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-06-15T09:55:40.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:40.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:40.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-06-15T09:55:40.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-06-15T09:55:40.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-06-15T09:55:40.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:40.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:41.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-06-15T09:55:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-06-15T09:55:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509470' 2024-06-15T09:55:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:41.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-06-15T09:55:41.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:41.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:41.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-06-15T09:55:41.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:41.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-06-15T09:55:41.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-06-15T09:55:41.082 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 171798691845 2024-06-15T09:55:41.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:41.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-06-15T09:55:41.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:41.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-06-15T09:55:41.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:41.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:41.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-06-15T09:55:41.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-06-15T09:55:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-06-15T09:55:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 193273528322 2024-06-15T09:55:41.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:41.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 193273528322 2024-06-15T09:55:41.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:42.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:55:42.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:42.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 193273528322 2024-06-15T09:55:42.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:55:43.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T09:55:43.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:44.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-06-15T09:55:44.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:44.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-06-15T09:55:44.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:44.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:44.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-06-15T09:55:44.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:44.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-06-15T09:55:44.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-06-15T09:55:44.070 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509470 2024-06-15T09:55:44.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:44.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-06-15T09:55:44.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:55:44.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:44.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:44.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:55:44.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:55:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:55:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:55:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:55:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:55:44.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:55:44.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:55:44.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:55:44.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:55:44.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:55:44.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:55:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:55:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:55:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:55:45.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-06-15T09:55:45.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-15T09:55:45.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-15T09:55:45.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:45.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:45.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:45.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-15T09:55:45.545 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T09:55:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:45.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:45.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:45.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:45.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:45.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:47.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:47.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:48.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:48.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:48.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:48.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:48.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:48.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:48.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:48.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:48.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:49.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:49.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:49.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:49.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:49.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:49.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:49.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:49.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:49.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:50.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:50.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:50.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:50.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:50.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:50.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:50.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:50.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:10.701860+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:50.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:55:51.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:55:51.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:55:51.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:55:51.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:55:51.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:55:51.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:55:51.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:55:52.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:55:46.319892+0000 '>' 2024-06-15T09:55:10.701860+0000 2024-06-15T09:55:52.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T09:55:52.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-06-15T09:55:54.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-06-15T09:55:54.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:55:54.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:55:54.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:55:54.269 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:55:54.269 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:55:54.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:55:54.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:54.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:55:54.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-06-15T09:55:54.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-06-15T09:55:54.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-06-15T09:55:54.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:54.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:55:54.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528326 2024-06-15T09:55:54.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528326 2024-06-15T09:55:54.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528326' 2024-06-15T09:55:54.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:55:54.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:55:54.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-06-15T09:55:54.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-06-15T09:55:54.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528326 2-64424509474' 2024-06-15T09:55:54.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:54.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-06-15T09:55:54.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:54.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:55:54.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-06-15T09:55:54.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:54.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-06-15T09:55:54.740 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 171798691849 2024-06-15T09:55:54.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-06-15T09:55:54.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:55:54.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-06-15T09:55:54.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:54.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528326 2024-06-15T09:55:54.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:54.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:55:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:54.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528326 2024-06-15T09:55:54.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528326 2024-06-15T09:55:54.957 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 193273528326 2024-06-15T09:55:54.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528326' 2024-06-15T09:55:54.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:55:55.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528326 -lt 193273528326 2024-06-15T09:55:55.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:55:55.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-06-15T09:55:55.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:55:55.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:55:55.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-06-15T09:55:55.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:55:55.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-06-15T09:55:55.186 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509474 2024-06-15T09:55:55.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-06-15T09:55:55.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:55:55.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-06-15T09:55:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-15T09:55:55.622 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:55:55.622 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-06-15T09:55:46.320108+0000 21'10 46:133 [1,0] 1 [1,0] 1 21'10 2024-06-15T09:55:46.319892+0000 21'10 2024-06-15T09:55:46.319892+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:55:46.319892+0000 9 0 2024-06-15T09:55:55.622 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:55:55.622 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:55:55.622 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:55:55.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-06-15T09:55:55.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-06-15T09:55:55.878 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:55:55.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T09:55:55.896 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:55.896 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.429+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:10.697+0000 7f87b749b640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:46.316+0000 7f8abc638640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:46.316+0000 7f8abc638640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T09:55:55.897 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:46.316+0000 7f8abc638640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-15T09:55:55.898 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:46.316+0000 7f8abc638640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-06-15T09:55:55.898 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T09:55:46.316+0000 7f8abc638640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T09:55:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T09:55:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:55:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:55:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:55:55.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:55:55.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:55:55.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:55:55.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:55:55.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:55:56.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:55:56.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:55:56.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:55:56.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:55:56.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:55:56.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:55:56.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:55:56.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:55:56.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:55:56.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:55:56.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:55:56.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:55:56.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:55:56.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:55:56.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:55:56.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:55:56.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:55:56.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:55:56.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:55:56.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:55:56.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:55:56.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:55:56.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:55:56.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:55:56.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:55:56.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:55:56.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:55:56.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:55:56.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:55:56.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:55:56.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:55:56.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:55:56.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:55:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:55:56.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:55:56.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:55:56.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:55:56.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:55:56.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:55:56.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T09:55:56.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:55:56.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-06-15T09:55:56.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T09:55:56.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T09:55:56.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T09:55:56.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:55:56.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:55:56.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:55:56.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:55:56.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:55:56.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:55:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:55:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:55:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:55:56.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:55:56.782 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:55:56.783 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:55:56.783 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:55:56.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:55:56.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T09:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:55:57.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:55:57.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:57.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:55:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:55:57.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:55:57.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:55:57.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:55:57.800 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:55:57.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:55:57.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:55:57.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:55:57.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:55:57.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 62537943-1249-45d7-a938-362ccec5d510' 2024-06-15T09:55:57.803 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:55:57.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:55:57.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtZW1mFwksMRAAmcO/PMsbNdorChxnmIEdag== 2024-06-15T09:55:57.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtZW1mFwksMRAAmcO/PMsbNdorChxnmIEdag=="}' 2024-06-15T09:55:57.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62537943-1249-45d7-a938-362ccec5d510 -i td/osd-scrub-repair/0/new.json 2024-06-15T09:55:58.075 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:55:58.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T09:55:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAtZW1mFwksMRAAmcO/PMsbNdorChxnmIEdag== --osd-uuid 62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:55:58.131 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.132+0000 7f344871c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:58.142 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.144+0000 7f344871c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:58.143 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.144+0000 7f344871c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:55:58.143 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.144+0000 7f344871c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:55:58.143 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.144+0000 7f344871c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:55:58.144 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.144+0000 7f344871c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:55:58.242 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:55:58.244+0000 7f344871c7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T09:56:04.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T09:56:04.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:56:04.459 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:56:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:56:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:56:04.718 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:56:04.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:56:04.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:56:04.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:56:04.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:56:04.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:56:04.753 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:04.752+0000 7f1a65f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:04.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:04.784+0000 7f1a65f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:04.782 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:04.784+0000 7f1a65f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:04.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:56:04.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:56:04.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:56:04.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:04.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:05.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:05.332 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:05.332+0000 7f1a65f1f7c0 -1 Falling back to public interface 2024-06-15T09:56:06.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:06.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:06.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:56:06.223 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:56:06.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:06.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:06.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:07.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:07.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:07.463 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:56:07.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:56:07.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:07.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:07.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:07.994 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:07.996+0000 7f1a65f1f7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:56:08.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:08.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:08.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:56:08.705 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:56:08.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:08.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:09.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:10.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:10.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:10.079 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:56:10.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:56:10.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:10.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:10.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:11.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:11.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:56:11.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:56:11.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:11.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:11.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:12.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:12.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:12.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:56:12.559 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:56:12.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:12.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:12.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:13.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:13.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:13.870 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:56:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:56:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:13.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:14.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:15.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:15.117 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:56:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:56:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:15.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:15.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:16.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:16.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:16.361 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:56:16.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:56:16.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:16.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:16.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:17.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:17.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:17.659 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:56:17.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:56:17.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:17.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:17.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:18.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:18.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:18.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:56:18.986 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:56:18.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:18.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:19.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:20.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:20.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:20.225 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:56:20.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:56:20.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:20.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:20.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:21.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:21.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:21.471 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:56:21.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:56:21.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:21.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:21.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:22.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:22.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:22.836 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:56:22.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:56:22.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:22.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:23.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:24.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:24.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:24.484 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:56:24.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:56:24.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:24.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:24.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:25.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:25.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:25.875 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:56:25.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:56:25.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:25.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:26.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:27.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:27.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:27.118 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:56:27.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:56:27.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:27.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:27.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:28.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:28.743 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:56:28.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:56:28.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:28.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:29.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:30.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:30.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:30.051 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:56:30.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:56:30.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:30.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:30.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:31.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:31.701 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:56:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:56:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:31.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:33.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:33.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:33.001 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:56:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:56:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:56:33.258 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31127195,v1:127.0.0.1:6803/31127195] [v2:127.0.0.1:6804/31127195,v1:127.0.0.1:6805/31127195] exists,up 62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:56:33.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:56:33.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:56:33.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:56:33.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:56:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:56:33.268 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:56:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c081bf2b-592b-4957-a03d-82611e92b889' 2024-06-15T09:56:33.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:56:33.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRZW1miSd3ERAA1+X+gXuRSY0REOK5czZxBA== 2024-06-15T09:56:33.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRZW1miSd3ERAA1+X+gXuRSY0REOK5czZxBA=="}' 2024-06-15T09:56:33.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c081bf2b-592b-4957-a03d-82611e92b889 -i td/osd-scrub-repair/1/new.json 2024-06-15T09:56:33.611 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:56:33.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T09:56:33.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBRZW1miSd3ERAA1+X+gXuRSY0REOK5czZxBA== --osd-uuid c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:56:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.668+0000 7ff43feaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:33.704 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.704+0000 7ff43feaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:33.705 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.708+0000 7ff43feaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:33.705 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.708+0000 7ff43feaa7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:56:33.706 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.708+0000 7ff43feaa7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:56:33.706 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.708+0000 7ff43feaa7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:56:33.777 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:33.780+0000 7ff43feaa7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T09:56:40.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T09:56:40.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:56:40.360 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:56:40.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:56:40.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:56:40.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:56:40.633 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:56:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:56:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:56:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:56:40.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:56:40.670 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:40.672+0000 7f049ab2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:40.699 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:40.700+0000 7f049ab2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:40.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:40.700+0000 7f049ab2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:40.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:56:40.891 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:56:40.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:40.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:41.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:41.260 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:41.260+0000 7f049ab2c7c0 -1 Falling back to public interface 2024-06-15T09:56:42.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:42.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:42.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:56:42.138 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:56:42.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:42.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:42.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:43.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:43.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:43.371 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:56:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:56:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:43.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:43.906 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:56:43.908+0000 7f049ab2c7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:56:44.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:44.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:44.612 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:56:44.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:56:44.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:44.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:45.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:46.031 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:56:46.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:56:46.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:46.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:47.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:47.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:47.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:56:47.274 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:56:47.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:47.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:47.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:48.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:48.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:48.564 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:56:48.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:56:48.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:48.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:48.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:49.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:49.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:49.807 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:56:49.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:56:49.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:49.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:50.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:51.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:51.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:51.190 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:56:51.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:56:51.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:51.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:51.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:52.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:52.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:52.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:56:52.474 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:56:52.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:52.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:52.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:53.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:53.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:53.715 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:56:53.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:56:53.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:53.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:54.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:55.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:55.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:55.066 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:56:55.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:56:55.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:55.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:55.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:56.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:56.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:56.392 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:56:56.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:56:56.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:56.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:56.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:57.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:57.630 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:56:57.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:56:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:57.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:58.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:56:59.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:56:59.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:56:59.053 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:56:59.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:56:59.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:56:59.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:56:59.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:00.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:00.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:00.720 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:57:00.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:57:00.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:00.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:00.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:01.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:01.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:01.980 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:57:01.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:57:01.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:01.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:03.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:04.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:04.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:04.034 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:57:04.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:57:04.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:04.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:04.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:05.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:05.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:05.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:57:05.490 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:57:05.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:05.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:06.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:07.268 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:57:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:57:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:08.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:09.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:09.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:09.087 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:57:09.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:57:09.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:09.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:57:09.421 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/13767861,v1:127.0.0.1:6811/13767861] [v2:127.0.0.1:6812/13767861,v1:127.0.0.1:6813/13767861] exists,up c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:57:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:57:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:57:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:57:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T09:57:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:57:09.425 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:57:09.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:57:09.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f63f38ad-d06c-4fa2-a2e6-94b1c30db772 2024-06-15T09:57:09.429 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 f63f38ad-d06c-4fa2-a2e6-94b1c30db772 2024-06-15T09:57:09.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f63f38ad-d06c-4fa2-a2e6-94b1c30db772' 2024-06-15T09:57:09.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:57:09.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1ZW1m04rpGhAA+Jc3Rngu9XA4U0v+1UkBAg== 2024-06-15T09:57:09.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1ZW1m04rpGhAA+Jc3Rngu9XA4U0v+1UkBAg=="}' 2024-06-15T09:57:09.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f63f38ad-d06c-4fa2-a2e6-94b1c30db772 -i td/osd-scrub-repair/2/new.json 2024-06-15T09:57:09.754 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:57:09.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T09:57:09.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB1ZW1m04rpGhAA+Jc3Rngu9XA4U0v+1UkBAg== --osd-uuid f63f38ad-d06c-4fa2-a2e6-94b1c30db772 2024-06-15T09:57:09.812 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.811+0000 7f907df057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:09.855 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.859+0000 7f907df057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:09.857 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.859+0000 7f907df057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:09.858 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.859+0000 7f907df057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:57:09.858 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.859+0000 7f907df057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:57:09.858 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.859+0000 7f907df057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T09:57:09.938 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:09.939+0000 7f907df057c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T09:57:16.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T09:57:16.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:57:16.584 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T09:57:16.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T09:57:16.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:57:16.861 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T09:57:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T09:57:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:57:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:57:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:57:16.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:57:16.897 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:16.899+0000 7f978f3dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:16.902 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:16.903+0000 7f978f3dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:16.904 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:16.903+0000 7f978f3dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:17.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T09:57:17.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:57:17.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T09:57:17.099 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:17.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:17.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:17.467 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:17.467+0000 7f978f3dc7c0 -1 Falling back to public interface 2024-06-15T09:57:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:57:18.370 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:57:18.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:18.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:18.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:19.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:19.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:19.611 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:57:19.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:57:19.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:19.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:19.621 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:19.623+0000 7f978f3dc7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T09:57:19.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:20.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:20.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:20.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:57:20.855 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:57:20.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:20.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:21.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:22.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:22.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:22.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:57:22.292 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:57:22.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:22.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:22.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:23.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:23.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:23.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:57:23.642 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:57:23.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:23.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:23.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:24.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:24.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:24.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:57:24.884 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:57:24.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:24.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:25.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:26.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:26.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:26.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:57:26.291 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:57:26.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:26.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:26.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:27.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:27.663 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:57:27.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:57:27.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:27.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:27.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:28.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:28.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:28.905 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:57:28.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:57:28.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:28.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:29.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:30.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:30.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:30.347 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:57:30.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:57:30.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:30.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:30.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:31.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:31.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:31.644 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:57:31.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:57:31.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:31.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:31.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:32.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:32.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:32.887 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:57:32.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:57:32.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:32.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:33.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:34.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:34.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:34.177 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:57:34.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:57:34.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:34.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:34.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:35.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:35.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:35.526 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:57:35.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:57:35.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:35.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:36.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:37.325 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:57:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:57:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:37.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:38.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:38.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:38.588 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:57:38.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:57:38.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:38.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:39.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:40.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:40.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:40.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:57:40.041 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:57:40.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:40.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:40.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:41.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:41.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:41.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:57:41.283 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:57:41.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:41.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:41.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:42.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:42.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:42.984 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:57:42.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:57:42.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:42.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:43.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:44.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:44.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:44.254 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:57:44.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:57:44.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:44.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:44.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:45.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:45.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:45.785 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:57:45.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:57:45.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:45.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T09:57:46.026 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4041313425,v1:127.0.0.1:6819/4041313425] [v2:127.0.0.1:6820/4041313425,v1:127.0.0.1:6821/4041313425] exists,up f63f38ad-d06c-4fa2-a2e6-94b1c30db772 2024-06-15T09:57:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:57:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:57:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:57:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-06-15T09:57:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-15T09:57:46.480 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T09:57:46.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T09:57:47.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-06-15T09:57:47.924 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T09:57:47.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-15T09:57:47.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:57:47.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:57:47.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:57:47.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:57:47.946 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:57:47.946 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:57:47.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:57:47.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:57:47.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:57:48.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:57:48.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:57:48.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:57:48.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:57:48.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:57:48.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:57:48.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:57:48.438 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:57:48.441 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:57:48.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:57:48.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:57:48.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:57:48.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-06-15T09:57:48.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-06-15T09:57:48.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-06-15T09:57:48.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:57:48.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:57:48.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-15T09:57:48.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-15T09:57:48.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-06-15T09:57:48.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:57:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:57:48.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-06-15T09:57:48.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-06-15T09:57:48.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509444' 2024-06-15T09:57:48.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:57:48.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-06-15T09:57:48.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:57:48.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:57:48.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-06-15T09:57:48.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:57:48.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-06-15T09:57:48.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-06-15T09:57:48.968 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836501 2024-06-15T09:57:48.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:57:49.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-06-15T09:57:49.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:57:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:57:50.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:57:50.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-06-15T09:57:50.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:57:50.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:57:50.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-15T09:57:50.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:57:50.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-15T09:57:50.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:57:50.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-15T09:57:50.449 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672972 2024-06-15T09:57:50.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-15T09:57:50.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:57:50.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672972 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-06-15T09:57:50.695 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509444 2024-06-15T09:57:50.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-06-15T09:57:50.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:57:50.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-06-15T09:57:50.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:57:50.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:57:50.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:57:51.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:57:51.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:57:51.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:57:51.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:57:51.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:57:51.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:57:51.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:57:51.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:57:51.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:57:51.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:57:51.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:57:51.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-06-15T09:57:51.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-06-15T09:57:51.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:51.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:51.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:51.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:52.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:52.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:53.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:53.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:53.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-06-15T09:57:53.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-15T09:57:53.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-15T09:57:53.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:57:53.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:57:53.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:57:53.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:57:53.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:57:53.437 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:57:53.437 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:57:53.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:57:53.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:57:53.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:57:53.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:57:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:57:54.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-06-15T09:57:55.891 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-15T09:57:56.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:57:56.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:57:56.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:57:56.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:57:56.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:57:56.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:57:56.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:57:56.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:57:56.432 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:57:56.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:57:56.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:57:56.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:57:56.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:57:56.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:57:56.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:57:56.473 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:56.475+0000 7fb0d03717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:56.476 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:56.475+0000 7fb0d03717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:56.477 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:56.479+0000 7fb0d03717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:57:56.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:57:56.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:57.011 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:57.011+0000 7fb0d03717c0 -1 Falling back to public interface 2024-06-15T09:57:57.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:57:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:57:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:57.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:57:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:59.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:57:59.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:57:59.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:57:59.178 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:57:59.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:57:59.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:57:59.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:57:59.748 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:57:59.751+0000 7fb0d03717c0 -1 osd.0 21 log_to_monitors true 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3116902869,v1:127.0.0.1:6803/3116902869] [v2:127.0.0.1:6804/3116902869,v1:127.0.0.1:6805/3116902869] exists,up 62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:58:00.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:58:00.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:58:00.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:58:00.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:58:00.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:58:00.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:58:00.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:58:00.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:58:00.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:58:00.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:01.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:01.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-15T09:58:01.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-15T09:58:01.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-15T09:58:01.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:01.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:01.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-15T09:58:01.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-15T09:58:01.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976' 2024-06-15T09:58:01.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:01.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-15T09:58:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-15T09:58:01.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672976 2-64424509448' 2024-06-15T09:58:01.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:01.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-15T09:58:01.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:01.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:01.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-15T09:58:01.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:01.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-15T09:58:01.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-15T09:58:01.477 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182402 2024-06-15T09:58:01.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-06-15T09:58:01.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:02.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:02.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:02.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-15T09:58:02.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:02.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-15T09:58:02.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:02.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:02.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-15T09:58:02.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:02.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-15T09:58:02.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-15T09:58:02.999 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672976 2024-06-15T09:58:02.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:03.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-15T09:58:03.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:03.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:03.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-15T09:58:03.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:03.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-15T09:58:03.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:03.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-15T09:58:03.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-15T09:58:03.277 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509448 2024-06-15T09:58:03.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:03.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-06-15T09:58:03.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:58:03.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:03.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:03.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:03.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:03.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:04.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:58:04.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:04.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:04.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:04.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:58:04.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T09:58:04.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T09:58:04.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T09:58:04.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:58:04.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T09:58:04.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:58:04.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:04.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:04.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:04.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:04.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:04.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:04.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:04.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:58:04.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:04.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:04.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:04.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:58:04.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T09:58:04.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T09:58:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T09:58:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T09:58:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T09:58:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:58:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T09:58:05.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:58:05.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:05.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:05.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T09:58:05.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:05.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:05.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T09:58:05.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T09:58:05.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T09:58:05.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T09:58:06.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T09:58:06.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T09:58:06.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T09:58:06.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T09:58:06.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T09:58:06.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T09:58:06.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:06.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:06.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:06.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:06.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:06.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:06.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:06.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:58:06.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:06.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:06.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:07.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:58:07.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:58:07.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:58:07.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-15T09:58:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:58:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:58:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:58:07.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:58:07.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T09:58:07.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:58:07.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:58:07.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:58:07.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:58:07.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:58:07.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-06-15T09:58:09.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:58:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:09.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:09.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:58:09.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:58:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T09:58:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:09.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T09:58:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:58:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:58:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:58:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:58:09.761 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:09.767+0000 7f50f99e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:09.764 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:09.767+0000 7f50f99e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:09.765 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:09.771+0000 7f50f99e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:58:09.972 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:58:09.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:09.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:10.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:10.346 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:10.351+0000 7f50f99e97c0 -1 Falling back to public interface 2024-06-15T09:58:11.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:11.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:11.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:58:11.286 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:11.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:11.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:11.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:12.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:12.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:12.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:58:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:58:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:12.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:12.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:12.970 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:12.975+0000 7f50f99e97c0 -1 osd.1 26 log_to_monitors true 2024-06-15T09:58:13.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:13.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:13.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:58:13.771 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:58:13.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:13.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:14.067 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/947781146,v1:127.0.0.1:6811/947781146] [v2:127.0.0.1:6812/947781146,v1:127.0.0.1:6813/947781146] exists,up c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:58:14.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:58:14.069 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:58:14.069 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:58:14.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:58:14.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:58:14.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:58:14.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:58:14.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:58:14.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:58:14.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:58:14.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:14.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:14.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:14.553 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:14.553 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:14.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:14.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:14.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:14.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-06-15T09:58:14.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-06-15T09:58:14.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-06-15T09:58:14.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:14.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:14.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T09:58:14.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T09:58:14.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-06-15T09:58:14.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:14.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-06-15T09:58:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-06-15T09:58:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509451' 2024-06-15T09:58:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:15.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-06-15T09:58:15.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:15.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:15.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-06-15T09:58:15.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:15.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-06-15T09:58:15.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-06-15T09:58:15.015 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182405 2024-06-15T09:58:15.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:15.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-06-15T09:58:15.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:16.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:16.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:16.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182405 2024-06-15T09:58:16.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:16.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-06-15T09:58:16.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:16.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:16.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-06-15T09:58:16.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:16.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T09:58:16.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-06-15T09:58:16.498 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 128849018882 2024-06-15T09:58:16.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:16.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T09:58:16.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:16.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-06-15T09:58:16.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:16.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:16.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-06-15T09:58:16.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:16.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-06-15T09:58:16.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-15T09:58:16.735 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509451 2024-06-15T09:58:16.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:16.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-06-15T09:58:16.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:58:16.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:16.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:17.252 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:17.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:17.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:58:17.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:17.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:17.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:17.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:58:17.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:58:17.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:58:17.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-06-15T09:58:17.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T09:58:17.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T09:58:17.738 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T09:58:17.738 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:58:17.738 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:58:17.738 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:58:17.738 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:58:17.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T09:58:17.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:58:17.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:58:18.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:58:18.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:58:18.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:58:18.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:58:18.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-15T09:58:18.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:58:18.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:58:18.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T09:58:18.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:58:18.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:58:18.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-15T09:58:19.965 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:104778fc:::obj2:head# 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:58:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:20.504 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:20.505 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:58:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:20.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T09:58:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T09:58:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:58:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:58:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:58:20.542 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:20.547+0000 7fab2d5c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:20.545 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:20.551+0000 7fab2d5c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:20.546 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:20.551+0000 7fab2d5c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:20.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T09:58:20.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:20.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:58:20.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:21.079 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:21.083+0000 7fab2d5c87c0 -1 Falling back to public interface 2024-06-15T09:58:21.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:21.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:21.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:58:21.984 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:21.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:21.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:58:22.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:23.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:23.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:58:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:58:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:58:23.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:23.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:23.751+0000 7fab2d5c87c0 -1 osd.0 31 log_to_monitors true 2024-06-15T09:58:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:58:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:58:24.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:24.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/182732554,v1:127.0.0.1:6803/182732554] [v2:127.0.0.1:6804/182732554,v1:127.0.0.1:6805/182732554] exists,up 62537943-1249-45d7-a938-362ccec5d510 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:58:24.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:58:24.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:58:24.735 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:58:24.735 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:58:24.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:58:24.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:58:24.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:24.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:25.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-15T09:58:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-15T09:58:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-06-15T09:58:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:25.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:25.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-06-15T09:58:25.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-06-15T09:58:25.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-06-15T09:58:25.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:25.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:25.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-06-15T09:58:25.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-06-15T09:58:25.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509454' 2024-06-15T09:58:25.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:25.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-06-15T09:58:25.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:25.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:25.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-06-15T09:58:25.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-15T09:58:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-06-15T09:58:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 150323855362 2024-06-15T09:58:25.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-06-15T09:58:25.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:26.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:26.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:27.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-15T09:58:27.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:27.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-06-15T09:58:27.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:27.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:27.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:27.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-06-15T09:58:27.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-06-15T09:58:27.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-06-15T09:58:27.142 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 128849018885 2024-06-15T09:58:27.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:27.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-06-15T09:58:27.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:27.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-06-15T09:58:27.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:27.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:27.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-06-15T09:58:27.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:27.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-06-15T09:58:27.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-15T09:58:27.419 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509454 2024-06-15T09:58:27.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:27.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-06-15T09:58:27.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:58:27.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:27.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:27.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:27.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:58:28.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:28.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:28.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:28.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:58:28.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:58:28.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:58:28.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-06-15T09:58:28.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:58:28.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T09:58:28.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T09:58:28.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:58:28.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:58:28.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:58:28.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:58:28.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:58:28.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T09:58:28.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:58:28.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-06-15T09:58:30.557 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-06-15T09:58:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T09:58:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T09:58:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T09:58:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T09:58:31.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:31.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:31.115 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:31.115 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:31.115 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:31.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:31.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T09:58:31.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:58:31.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T09:58:31.117 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T09:58:31.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T09:58:31.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T09:58:31.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T09:58:31.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T09:58:31.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T09:58:31.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T09:58:31.159 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:31.163+0000 7f40359027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:31.162 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:31.167+0000 7f40359027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:31.163 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:31.167+0000 7f40359027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T09:58:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:58:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:58:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:58:31.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:58:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:58:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T09:58:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:31.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:31.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:31.703 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:31.707+0000 7f40359027c0 -1 Falling back to public interface 2024-06-15T09:58:32.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:32.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:32.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:58:32.606 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:32.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:32.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:32.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:33.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:33.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:33.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:58:33.850 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T09:58:33.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:33.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:33.995 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:33.998+0000 7f40359027c0 -1 osd.1 36 log_to_monitors true 2024-06-15T09:58:34.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:58:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:58:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:58:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:58:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T09:58:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:58:35.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/3135528889,v1:127.0.0.1:6811/3135528889] [v2:127.0.0.1:6812/3135528889,v1:127.0.0.1:6813/3135528889] exists,up c081bf2b-592b-4957-a03d-82611e92b889 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T09:58:35.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:58:35.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:58:35.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:58:35.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:35.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:35.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:35.691 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:35.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-06-15T09:58:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-06-15T09:58:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-06-15T09:58:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:35.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T09:58:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T09:58:36.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-06-15T09:58:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:36.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-06-15T09:58:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-06-15T09:58:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509458' 2024-06-15T09:58:36.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:36.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-06-15T09:58:36.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:36.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:36.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-06-15T09:58:36.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:36.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-06-15T09:58:36.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-06-15T09:58:36.181 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 150323855365 2024-06-15T09:58:36.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:36.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-06-15T09:58:36.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:36.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-06-15T09:58:36.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:36.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:36.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-06-15T09:58:36.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:36.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T09:58:36.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-06-15T09:58:36.427 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 171798691842 2024-06-15T09:58:36.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:36.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-15T09:58:36.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:36.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-06-15T09:58:36.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:36.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:36.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-06-15T09:58:36.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:36.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-06-15T09:58:36.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-15T09:58:36.694 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509458 2024-06-15T09:58:36.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:36.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509458 2024-06-15T09:58:36.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:37.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:37.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:38.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-15T09:58:38.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:58:38.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:38.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:38.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:58:38.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:38.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:38.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:58:38.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:38.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:38.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:39.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:58:39.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:58:39.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:58:39.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-06-15T09:58:39.139 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: { 2024-06-15T09:58:39.140 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-15T09:58:39.140 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:58:39.153 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: { 2024-06-15T09:58:39.153 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-06-15T09:58:39.153 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:58:39.166 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2: { 2024-06-15T09:58:39.166 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-06-15T09:58:39.167 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:58:39.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-06-15T09:58:39.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T09:58:39.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-15T09:58:39.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-15T09:58:39.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T09:58:39.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-06-15T09:58:39.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-06-15T09:58:39.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T09:58:39.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T09:58:39.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T09:58:39.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T09:58:39.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-06-15T09:58:39.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-06-15T09:58:39.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:39.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:39.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:39.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:39.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:39.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-06-15T09:58:40.086 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T09:58:40.086 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": true, 2024-06-15T09:58:40.086 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T09:58:40.086 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-01T09:57:00.092928+0000" 2024-06-15T09:58:40.086 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:40.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:40.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:40.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:40.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:57:46.343502+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:40.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:58:41.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:58:41.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:41.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:41.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:41.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:41.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:41.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:41.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:57:46.343502+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:41.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:58:42.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:58:42.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:42.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:42.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:42.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:42.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:42.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:42.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:57:46.343502+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:42.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:58:43.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:58:43.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:43.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:43.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:43.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:43.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:43.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:44.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:57:46.343502+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:44.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:58:45.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:58:45.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:45.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:45.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:45.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:45.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:45.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:45.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:57:46.343502+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:45.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T09:58:46.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T09:58:46.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T09:58:46.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T09:58:46.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T09:58:46.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T09:58:46.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T09:58:46.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T09:58:46.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T09:58:42.046725+0000 '>' 2024-06-15T09:57:46.343502+0000 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T09:58:46.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T09:58:46.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T09:58:46.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T09:58:46.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T09:58:46.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T09:58:46.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T09:58:46.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T09:58:46.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T09:58:46.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:46.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:46.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:46.855 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:46.855 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:46.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:46.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:46.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-06-15T09:58:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-06-15T09:58:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368' 2024-06-15T09:58:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:47.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-06-15T09:58:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-06-15T09:58:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-06-15T09:58:47.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:47.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-06-15T09:58:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-06-15T09:58:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509461' 2024-06-15T09:58:47.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:47.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855368 2024-06-15T09:58:47.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:47.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:47.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855368 2024-06-15T09:58:47.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:47.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-06-15T09:58:47.275 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 150323855368 2024-06-15T09:58:47.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-06-15T09:58:47.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:47.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855368 2024-06-15T09:58:47.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:48.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:48.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:48.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-06-15T09:58:48.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:48.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-06-15T09:58:48.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:48.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:48.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-06-15T09:58:48.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:48.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-06-15T09:58:48.735 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 171798691845 2024-06-15T09:58:48.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-06-15T09:58:48.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:49.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-06-15T09:58:49.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:49.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-06-15T09:58:49.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:49.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:49.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-06-15T09:58:49.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:49.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-06-15T09:58:49.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-15T09:58:49.018 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509461 2024-06-15T09:58:49.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:49.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-15T09:58:49.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T09:58:49.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:49.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T09:58:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T09:58:49.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T09:58:49.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T09:58:49.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T09:58:49.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T09:58:49.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T09:58:49.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T09:58:49.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T09:58:49.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T09:58:49.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T09:58:49.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T09:58:50.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T09:58:50.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T09:58:50.331 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T09:58:50.331 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T09:58:50.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T09:58:50.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:50.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T09:58:50.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-06-15T09:58:50.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-06-15T09:58:50.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370' 2024-06-15T09:58:50.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:50.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T09:58:50.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691847 2024-06-15T09:58:50.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691847 2024-06-15T09:58:50.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-06-15T09:58:50.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T09:58:50.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T09:58:50.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-06-15T09:58:50.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-06-15T09:58:50.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509462' 2024-06-15T09:58:50.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:50.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855370 2024-06-15T09:58:50.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:50.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T09:58:50.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855370 2024-06-15T09:58:50.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:50.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-06-15T09:58:50.783 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 150323855370 2024-06-15T09:58:50.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-06-15T09:58:50.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:51.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855370 2024-06-15T09:58:51.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T09:58:52.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T09:58:52.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T09:58:52.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-06-15T09:58:52.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:52.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691847 2024-06-15T09:58:52.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:52.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T09:58:52.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691847 2024-06-15T09:58:52.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:52.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691847 2024-06-15T09:58:52.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-06-15T09:58:52.269 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 171798691847 2024-06-15T09:58:52.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T09:58:52.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691847 2024-06-15T09:58:52.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T09:58:52.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-06-15T09:58:52.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T09:58:52.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T09:58:52.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-06-15T09:58:52.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T09:58:52.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-06-15T09:58:52.501 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509462 2024-06-15T09:58:52.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-15T09:58:52.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T09:58:52.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509462 2024-06-15T09:58:52.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-06-15T09:58:52.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-15T09:58:52.977 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T09:58:52.977 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-06-15T09:58:42.047005+0000 21'10 41:93 [1,0] 1 [1,0] 1 21'10 2024-06-15T09:58:42.046725+0000 21'10 2024-06-15T09:58:42.046725+0000 0 1 periodic scrub scheduled @ 2024-06-16T09:58:42.046725+0000 10 0 2024-06-15T09:58:52.977 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T09:58:52.977 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T09:58:52.977 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:58:52.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-06-15T09:58:52.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-06-15T09:58:53.190 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T09:58:53.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T09:58:53.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:58:53.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:58:53.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:58:53.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:58:53.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:58:53.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:58:53.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:58:53.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:58:53.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:58:53.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:58:53.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:58:53.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:58:53.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:58:53.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:58:53.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:58:53.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:58:53.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:58:53.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:58:53.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:58:53.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:58:53.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:58:53.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:58:53.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:58:53.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:53.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:53.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:58:53.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:58:53.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:58:53.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T09:58:53.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T09:58:53.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T09:58:53.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T09:58:53.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T09:58:53.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T09:58:53.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T09:58:53.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T09:58:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T09:58:53.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T09:58:53.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:53.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T09:58:53.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T09:58:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T09:58:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T09:58:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T09:58:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T09:58:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T09:58:53.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T09:58:53.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:53.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:53.765 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:53.765 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:53.765 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:53.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:53.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T09:58:54.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T09:58:54.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T09:58:54.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:58:54.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:58:54.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T09:58:54.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T09:58:54.012 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:58:54.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T09:58:54.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T09:58:54.130 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T09:58:54.131 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T09:58:54.131 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T09:58:54.131 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:54.132 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:54.132 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T09:58:54.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T09:58:54.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T09:58:54.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T09:58:55.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T09:58:55.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:55.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:55.152 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:55.152 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:55.152 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:55.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:55.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:58:55.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T09:58:55.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T09:58:55.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-06-15T09:58:55.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-15T09:58:55.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T09:58:55.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T09:58:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:58:55.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:58:55.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T09:58:55.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T09:58:55.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:58:55.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T09:58:55.197 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T09:58:55.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0223527a-db1f-41ad-9580-70a660434fed' 2024-06-15T09:58:55.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:58:55.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfZW1m0MvsDBAA6ndTSIXnH1inHEJgEOX/DQ== 2024-06-15T09:58:55.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfZW1m0MvsDBAA6ndTSIXnH1inHEJgEOX/DQ=="}' 2024-06-15T09:58:55.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0223527a-db1f-41ad-9580-70a660434fed -i td/osd-scrub-repair/0/new.json 2024-06-15T09:58:55.487 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:58:55.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T09:58:55.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDfZW1m0MvsDBAA6ndTSIXnH1inHEJgEOX/DQ== --osd-uuid 0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T09:58:55.544 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.550+0000 7f78eba0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:55.554 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.558+0000 7f78eba0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:55.555 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.558+0000 7f78eba0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:58:55.555 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.558+0000 7f78eba0f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:58:55.556 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.558+0000 7f78eba0f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:58:55.556 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.558+0000 7f78eba0f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T09:58:55.646 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:58:55.650+0000 7f78eba0f7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T09:59:03.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T09:59:03.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:59:03.315 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T09:59:03.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T09:59:03.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:59:03.579 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T09:59:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T09:59:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:59:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T09:59:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:59:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:59:03.613 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:03.618+0000 7fefbbe8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:03.661 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:03.666+0000 7fefbbe8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:03.663 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:03.666+0000 7fefbbe8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:03.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T09:59:03.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:03.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:59:03.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:03.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:04.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:04.219 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:04.222+0000 7fefbbe8d7c0 -1 Falling back to public interface 2024-06-15T09:59:05.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:59:05.046 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:59:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:05.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:06.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:06.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:06.271 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:59:06.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:59:06.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:06.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:06.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:06.866+0000 7fefbbe8d7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T09:59:07.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:07.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:07.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:59:07.513 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:59:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:07.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:08.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:08.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:08.895 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:59:08.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:59:08.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:08.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:09.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:10.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:10.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:10.139 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:59:10.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:59:10.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:10.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:10.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:11.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:11.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:11.380 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:59:11.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:59:11.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:11.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:11.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:12.634 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:59:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:59:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:13.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:14.061 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:59:14.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:59:14.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:14.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:14.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:15.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:15.307 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:59:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:59:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:15.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:15.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:16.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:16.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:16.552 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:59:16.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:59:16.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:16.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:16.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:17.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:17.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:17.931 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T09:59:17.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T09:59:17.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:17.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:18.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:19.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:19.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:19.173 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T09:59:19.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T09:59:19.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:19.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:19.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:20.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T09:59:20.505 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T09:59:20.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:20.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:20.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:21.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:21.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:21.744 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T09:59:21.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T09:59:21.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:21.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:22.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:23.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:23.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:23.042 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T09:59:23.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T09:59:23.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:23.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:24.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:24.280 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T09:59:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T09:59:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:24.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:25.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:25.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:25.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T09:59:25.769 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T09:59:25.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:25.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:27.431 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T09:59:27.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T09:59:27.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:27.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:28.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:29.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:29.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:29.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T09:59:29.072 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T09:59:29.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:29.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:30.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:30.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:30.693 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T09:59:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T09:59:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:31.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:32.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:32.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:32.106 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T09:59:32.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T09:59:32.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:32.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:32.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:33.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:33.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:33.746 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T09:59:33.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T09:59:33.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:33.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:34.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:35.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:35.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:35.378 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T09:59:35.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T09:59:35.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:35.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:36.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:37.038 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T09:59:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T09:59:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T09:59:37.331 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3258807505,v1:127.0.0.1:6803/3258807505] [v2:127.0.0.1:6804/3258807505,v1:127.0.0.1:6805/3258807505] exists,up 0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T09:59:37.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T09:59:37.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T09:59:37.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T09:59:37.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T09:59:37.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T09:59:37.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T09:59:37.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T09:59:37.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0411235-d935-45fa-8a61-826f58b8ddc5 2024-06-15T09:59:37.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d0411235-d935-45fa-8a61-826f58b8ddc5' 2024-06-15T09:59:37.339 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 d0411235-d935-45fa-8a61-826f58b8ddc5 2024-06-15T09:59:37.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T09:59:37.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJZm1mSPrKFRAAnVzBVhLy8arKEupeB+tnQQ== 2024-06-15T09:59:37.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJZm1mSPrKFRAAnVzBVhLy8arKEupeB+tnQQ=="}' 2024-06-15T09:59:37.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0411235-d935-45fa-8a61-826f58b8ddc5 -i td/osd-scrub-repair/1/new.json 2024-06-15T09:59:37.720 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:59:37.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T09:59:37.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQAJZm1mSPrKFRAAnVzBVhLy8arKEupeB+tnQQ== --osd-uuid d0411235-d935-45fa-8a61-826f58b8ddc5 2024-06-15T09:59:37.777 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.782+0000 7f01fab507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:37.846 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.850+0000 7f01fab507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:37.848 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.854+0000 7f01fab507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:37.849 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.854+0000 7f01fab507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:59:37.849 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.854+0000 7f01fab507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:59:37.849 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.854+0000 7f01fab507c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T09:59:37.952 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:37.954+0000 7f01fab507c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T09:59:46.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T09:59:46.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T09:59:46.158 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T09:59:46.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T09:59:46.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T09:59:46.434 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T09:59:46.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T09:59:46.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T09:59:46.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T09:59:46.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T09:59:46.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T09:59:46.469 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:46.474+0000 7fd5fede27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:46.471 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:46.474+0000 7fd5fede27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:46.472 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:46.478+0000 7fd5fede27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T09:59:46.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:46.680 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T09:59:46.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T09:59:46.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:46.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:46.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:47.050 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:47.054+0000 7fd5fede27c0 -1 Falling back to public interface 2024-06-15T09:59:47.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:47.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:47.913 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T09:59:47.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T09:59:47.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:47.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:48.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:49.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:49.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:49.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T09:59:49.155 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T09:59:49.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:49.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:49.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:49.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T09:59:49.970+0000 7fd5fede27c0 -1 osd.1 0 log_to_monitors true 2024-06-15T09:59:50.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:50.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:50.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T09:59:50.391 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T09:59:50.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:50.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:50.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:51.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:51.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:51.870 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T09:59:51.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T09:59:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:51.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:52.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:53.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:53.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:53.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T09:59:53.187 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T09:59:53.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:53.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:53.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:54.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:54.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:54.429 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T09:59:54.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T09:59:54.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:54.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:54.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:55.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:55.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:55.901 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T09:59:55.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T09:59:55.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:55.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:56.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:57.234 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T09:59:57.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T09:59:57.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:57.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:57.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:58.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:58.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:58.576 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T09:59:58.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T09:59:58.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:58.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T09:59:58.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T09:59:59.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T09:59:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T09:59:59.996 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T09:59:59.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T09:59:59.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T09:59:59.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:00.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:01.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:01.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:01.378 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:00:01.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:00:01.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:01.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:01.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:02.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:02.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:02.635 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:00:02.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:00:02.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:02.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:02.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:03.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:03.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:03.911 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:00:03.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:00:03.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:03.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:04.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:05.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:05.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:05.201 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:00:05.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:00:05.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:05.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:05.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:06.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:06.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:06.441 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:00:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:00:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:06.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:07.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:07.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:07.951 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:00:07.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:00:07.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:07.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:08.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:09.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:09.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:09.191 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:00:09.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:00:09.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:09.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:09.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:10.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:10.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:10.832 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:00:10.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:00:10.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:10.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:11.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:12.175 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:00:12.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:00:12.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:12.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:12.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:13.839 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:00:13.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:00:13.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:13.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:15.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:15.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:15.491 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:00:15.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:00:15.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:15.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:16.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:17.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:17.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:17.156 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:00:17.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:00:17.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:17.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:17.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:18.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:18.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:18.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:00:18.949 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:00:18.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:18.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:19.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:20.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:20.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:20.189 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:00:20.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:00:20.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:20.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:20.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:21.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:21.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:21.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:00:21.839 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:00:21.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:21.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:00:22.080 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2487509468,v1:127.0.0.1:6811/2487509468] [v2:127.0.0.1:6812/2487509468,v1:127.0.0.1:6813/2487509468] exists,up d0411235-d935-45fa-8a61-826f58b8ddc5 2024-06-15T10:00:22.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:00:22.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:00:22.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:00:22.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:00:22.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:00:22.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=420ca40c-4838-4fbb-a28d-67fb1768a6cf 2024-06-15T10:00:22.087 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 420ca40c-4838-4fbb-a28d-67fb1768a6cf 2024-06-15T10:00:22.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 420ca40c-4838-4fbb-a28d-67fb1768a6cf' 2024-06-15T10:00:22.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:00:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2Zm1m8HvHBhAArs3wRNB3IM5DkYfhKP2Png== 2024-06-15T10:00:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2Zm1m8HvHBhAArs3wRNB3IM5DkYfhKP2Png=="}' 2024-06-15T10:00:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 420ca40c-4838-4fbb-a28d-67fb1768a6cf -i td/osd-scrub-repair/2/new.json 2024-06-15T10:00:22.517 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:00:22.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:00:22.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQA2Zm1m8HvHBhAArs3wRNB3IM5DkYfhKP2Png== --osd-uuid 420ca40c-4838-4fbb-a28d-67fb1768a6cf 2024-06-15T10:00:22.580 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.581+0000 7f1a99a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:22.652 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.657+0000 7f1a99a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:22.654 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.657+0000 7f1a99a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:22.654 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.657+0000 7f1a99a857c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:00:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.657+0000 7f1a99a857c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:00:22.655 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.657+0000 7f1a99a857c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:00:22.743 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:22.749+0000 7f1a99a857c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:00:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:00:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:00:30.934 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:00:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:00:30.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:00:31.204 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:00:31.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:00:31.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:00:31.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:00:31.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:00:31.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:00:31.239 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:31.241+0000 7f2f496da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:31.284 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:31.289+0000 7f2f496da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:31.285 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:31.289+0000 7f2f496da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:00:31.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:31.439 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:00:31.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:00:31.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:31.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:31.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:31.862 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:31.865+0000 7f2f496da7c0 -1 Falling back to public interface 2024-06-15T10:00:32.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:32.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:32.709 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:00:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:00:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:32.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:33.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:33.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:33.949 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:00:33.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:00:33.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:33.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:34.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:00:34.769+0000 7f2f496da7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:00:35.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:35.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:35.189 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:00:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:00:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:35.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:36.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:36.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:36.824 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:00:36.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:00:36.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:36.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:37.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:38.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:38.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:38.176 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:00:38.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:00:38.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:38.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:38.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:39.580 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:00:39.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:00:39.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:39.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:40.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:41.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:41.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:41.008 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:00:41.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:00:41.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:41.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:41.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:42.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:42.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:42.263 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:00:42.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:00:42.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:42.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:42.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:43.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:43.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:43.690 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:00:43.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:00:43.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:43.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:44.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:45.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:45.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:45.029 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:00:45.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:00:45.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:45.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:45.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:46.271 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:00:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:00:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:46.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:47.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:47.646 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:00:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:00:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:48.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:49.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:49.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:49.035 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:00:49.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:00:49.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:49.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:49.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:50.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:50.280 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:00:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:00:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:50.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:51.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:51.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:51.522 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:00:51.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:00:51.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:51.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:52.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:53.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:53.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:53.040 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:00:53.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:00:53.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:53.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:53.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:54.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:54.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:54.417 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:00:54.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:00:54.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:54.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:54.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:55.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:55.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:55.929 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:00:55.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:00:55.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:55.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:56.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:57.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:57.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:57.869 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:00:57.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:00:57.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:57.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:00:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:00:59.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:00:59.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:00:59.585 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:00:59.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:00:59.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:00:59.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:01:00.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:01.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:01.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:01.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:01:01.492 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:01:01.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:01.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:01:01.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:02.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:02.854 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:01:02.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:01:02.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:02.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:01:03.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:04.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:04.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:04.515 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:01:04.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:01:04.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:04.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:01:05.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:06.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:06.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:06.007 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:01:06.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:01:06.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:06.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:01:06.253 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1575560170,v1:127.0.0.1:6819/1575560170] [v2:127.0.0.1:6820/1575560170,v1:127.0.0.1:6821/1575560170] exists,up 420ca40c-4838-4fbb-a28d-67fb1768a6cf 2024-06-15T10:01:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:01:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:01:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:01:06.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-06-15T10:01:06.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-15T10:01:06.651 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T10:01:06.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:01:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-06-15T10:01:08.408 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T10:01:08.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-15T10:01:08.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:01:08.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:01:08.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:01:08.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:01:08.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:01:08.436 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:01:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:01:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:01:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:01:08.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:01:08.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:01:08.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:01:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:01:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:01:08.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:01:09.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:01:09.036 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:09.036 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:01:09.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:01:09.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:09.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:01:09.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-06-15T10:01:09.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-06-15T10:01:09.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-06-15T10:01:09.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:09.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:01:09.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-06-15T10:01:09.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-06-15T10:01:09.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979' 2024-06-15T10:01:09.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:09.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:01:09.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-15T10:01:09.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-15T10:01:09.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-64424509447' 2024-06-15T10:01:09.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:09.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-06-15T10:01:09.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:09.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:01:09.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-06-15T10:01:09.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:09.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-06-15T10:01:09.571 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836503 2024-06-15T10:01:09.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-06-15T10:01:09.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:09.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-06-15T10:01:09.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:09.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:09.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-06-15T10:01:09.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:01:09.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-06-15T10:01:09.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:09.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-06-15T10:01:09.807 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672979 2024-06-15T10:01:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-15T10:01:09.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:01:10.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-06-15T10:01:10.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:10.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-15T10:01:10.044 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:10.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:01:10.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-15T10:01:10.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:10.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-15T10:01:10.047 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509447 2024-06-15T10:01:10.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-15T10:01:10.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:10.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-06-15T10:01:10.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:11.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:01:11.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:11.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-06-15T10:01:11.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:12.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:01:12.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:12.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-06-15T10:01:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:01:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:12.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:13.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:01:13.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:13.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:13.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:13.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:13.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:13.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:13.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:13.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:01:13.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:13.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:13.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:01:13.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:01:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:01:13.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:13.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:13.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:13.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:13.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:13.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:13.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:13.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:01:13.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:13.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:13.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:14.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:01:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:01:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:01:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-06-15T10:01:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-06-15T10:01:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:01:14.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-15T10:01:14.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T10:01:14.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:01:14.323 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T10:01:14.323 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:01:14.323 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:01:14.324 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:14.324 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:01:14.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:01:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:01:14.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:01:14.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:01:14.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:01:14.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:01:14.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:01:16.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:01:16.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-15T10:01:17.871 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T10:01:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:01:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:01:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:01:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:01:18.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:01:18.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:01:18.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:01:18.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:01:18.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:01:18.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:01:18.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T10:01:18.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:01:18.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:01:18.414 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:01:18.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:01:18.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:01:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:01:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:01:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:01:18.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:01:18.450 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:18.453+0000 7fb8d408c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:18.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:18.457+0000 7fb8d408c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:18.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:18.457+0000 7fb8d408c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:18.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:01:18.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:01:18.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:01:18.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:18.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:18.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:18.993+0000 7fb8d408c7c0 -1 Falling back to public interface 2024-06-15T10:01:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:01:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:19.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:19.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:20.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:21.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:21.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:21.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:01:21.132 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:01:21.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:21.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:21.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:22.111 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:22.113+0000 7fb8d408c7c0 -1 osd.0 19 log_to_monitors true 2024-06-15T10:01:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:01:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:01:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:22.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:22.822 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6802/2611914380,v1:127.0.0.1:6803/2611914380] [v2:127.0.0.1:6804/2611914380,v1:127.0.0.1:6805/2611914380] exists,up 0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:01:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:01:22.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:01:22.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:01:22.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:01:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:01:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:01:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:01:22.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:01:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:01:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:01:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:01:23.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:23.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:01:23.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-06-15T10:01:23.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-06-15T10:01:23.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-06-15T10:01:23.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:23.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:01:23.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-06-15T10:01:23.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-06-15T10:01:23.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-42949672983' 2024-06-15T10:01:23.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:23.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:01:23.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-06-15T10:01:23.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-06-15T10:01:23.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-42949672983 2-64424509451' 2024-06-15T10:01:23.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:23.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-06-15T10:01:23.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:23.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:01:23.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-06-15T10:01:23.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:23.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-06-15T10:01:23.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-06-15T10:01:23.894 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 98784247810 2024-06-15T10:01:23.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:24.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-06-15T10:01:24.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:25.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:01:25.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-06-15T10:01:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:26.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:01:26.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:26.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-06-15T10:01:26.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:26.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-06-15T10:01:26.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:26.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:01:26.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:26.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-06-15T10:01:26.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-06-15T10:01:26.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-06-15T10:01:26.609 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672983 2024-06-15T10:01:26.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:01:26.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-06-15T10:01:26.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:26.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-06-15T10:01:26.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:26.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:01:26.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-06-15T10:01:26.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:26.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-06-15T10:01:26.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-15T10:01:26.850 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509451 2024-06-15T10:01:26.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:27.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-06-15T10:01:27.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:01:27.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:27.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:27.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:01:27.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:27.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:27.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:01:27.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:27.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:27.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:27.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:01:27.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:01:27.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:01:27.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-06-15T10:01:27.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T10:01:27.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:01:27.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:27.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:01:28.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-06-15T10:01:28.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-06-15T10:01:28.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:01:28.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:01:28.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:28.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:01:28.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-06-15T10:01:28.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-06-15T10:01:28.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:28.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:28.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:28.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:28.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:28.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-06-15T10:01:28.710 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T10:01:28.710 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": false, 2024-06-15T10:01:28.710 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T10:01:28.710 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-08T09:59:48.717879+0000" 2024-06-15T10:01:28.710 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:01:28.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:01:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T10:01:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:28.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:28.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:01:06.565744+0000 '>' 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:28.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:01:29.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:01:29.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:01:29.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T10:01:29.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:29.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:29.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:29.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:30.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:01:06.565744+0000 '>' 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:30.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:01:31.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:01:31.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:01:31.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T10:01:31.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:31.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:31.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:31.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:31.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:01:06.565744+0000 '>' 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:31.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:01:32.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:01:32.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:01:32.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T10:01:32.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:32.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:32.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:32.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:32.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:01:06.565744+0000 '>' 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:32.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:01:33.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:01:33.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:01:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T10:01:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:01:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:01:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:01:33.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:01:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:01:30.108833+0000 '>' 2024-06-15T10:01:06.565744+0000 2024-06-15T10:01:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:01:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-15T10:01:34.118 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T10:01:34.119 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T10:01:30.484800+0000 19'1 24:63 [1,0] 1 [1,0] 1 19'1 2024-06-15T10:01:30.108833+0000 19'1 2024-06-15T10:01:30.108833+0000 0 1 periodic scrub scheduled @ 2024-06-16T10:01:30.108833+0000 1 0 2024-06-15T10:01:34.119 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T10:01:34.119 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T10:01:34.119 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T10:01:34.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-15T10:01:36.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-15T10:01:36.357 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T10:01:36.357 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T10:01:30.484800+0000 19'1 24:63 [1,0] 1 [1,0] 1 19'1 2024-06-15T10:01:30.108833+0000 19'1 2024-06-15T10:01:30.108833+0000 0 1 periodic scrub scheduled @ 2024-06-16T10:01:30.108833+0000 1 0 2024-06-15T10:01:36.357 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T10:01:36.357 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T10:01:36.357 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T10:01:36.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-06-15T10:01:38.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-15T10:01:38.593 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T10:01:38.593 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T10:01:30.484800+0000 19'1 24:63 [1,0] 1 [1,0] 1 19'1 2024-06-15T10:01:30.108833+0000 19'1 2024-06-15T10:01:30.108833+0000 0 1 periodic scrub scheduled @ 2024-06-16T10:01:30.108833+0000 1 0 2024-06-15T10:01:38.593 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T10:01:38.593 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T10:01:38.593 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T10:01:38.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-06-15T10:01:43.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-06-15T10:01:43.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:01:43.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:01:43.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:01:43.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:01:43.612 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:01:43.612 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:01:43.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:01:43.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:01:43.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:01:43.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:01:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:01:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:01:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:01:43.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:01:43.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:01:43.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:01:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:01:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:01:43.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:43.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:01:44.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247815 2024-06-15T10:01:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247815 2024-06-15T10:01:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815' 2024-06-15T10:01:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:44.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:01:44.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-06-15T10:01:44.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-06-15T10:01:44.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815 1-42949672988' 2024-06-15T10:01:44.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:44.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:01:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-06-15T10:01:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-06-15T10:01:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815 1-42949672988 2-64424509456' 2024-06-15T10:01:44.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:44.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247815 2024-06-15T10:01:44.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:44.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:01:44.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247815 2024-06-15T10:01:44.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:44.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247815 2024-06-15T10:01:44.499 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 98784247815 2024-06-15T10:01:44.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2024-06-15T10:01:44.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:44.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247815 2024-06-15T10:01:44.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:45.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:01:45.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:45.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247815 -lt 98784247815 2024-06-15T10:01:45.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:45.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-06-15T10:01:45.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:45.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:01:45.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-06-15T10:01:45.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:45.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-06-15T10:01:45.978 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672988 2024-06-15T10:01:45.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-06-15T10:01:45.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:01:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-06-15T10:01:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:46.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-06-15T10:01:46.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:46.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:01:46.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-06-15T10:01:46.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:46.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-06-15T10:01:46.261 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509456 2024-06-15T10:01:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-06-15T10:01:46.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:46.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-06-15T10:01:46.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:01:46.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:46.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:46.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:01:46.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:46.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:46.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:46.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:46.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:46.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:46.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:47.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:01:47.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:47.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:47.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:47.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:01:47.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:01:47.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:01:47.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-06-15T10:01:47.532 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T10:01:47.532 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T10:01:30.484800+0000 19'1 24:63 [1,0] 1 [1,0] 1 19'1 2024-06-15T10:01:30.108833+0000 19'1 2024-06-15T10:01:30.108833+0000 0 1 periodic scrub scheduled @ 2024-06-16T10:01:30.108833+0000 1 0 2024-06-15T10:01:47.532 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T10:01:47.532 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T10:01:47.532 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T10:01:47.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-06-15T10:01:47.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T10:01:47.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:01:47.549 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T10:01:47.549 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:01:47.549 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:01:47.549 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:47.549 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:01:47.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:01:47.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:01:47.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:01:48.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:01:48.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:01:48.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:01:48.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:01:48.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:01:48.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:01:48.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:01:48.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:01:48.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:01:48.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:01:48.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:01:48.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:01:48.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:01:48.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:01:48.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:01:48.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:01:48.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-15T10:01:50.113 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:01:50.113 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:01:50.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:01:50.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:01:50.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-06-15T10:01:50.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:01:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:01:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:01:50.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-06-15T10:01:50.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:01:50.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:01:50.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:01:50.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:01:50.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:01:50.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:50.456+0000 7fc0518267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:50.457 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:50.460+0000 7fc0518267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:50.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:50.460+0000 7fc0518267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:01:50.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:01:50.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:50.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:50.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:51.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:51.008+0000 7fc0518267c0 -1 Falling back to public interface 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:51.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:53.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:53.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:01:53.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:01:53.804+0000 7fc0518267c0 -1 osd.0 24 log_to_monitors true 2024-06-15T10:01:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:01:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:01:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:01:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:01:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:01:54.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:01:54.656 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 25 last_clean_interval [23,24) [v2:127.0.0.1:6802/3152731033,v1:127.0.0.1:6803/3152731033] [v2:127.0.0.1:6804/3152731033,v1:127.0.0.1:6805/3152731033] exists,up 0223527a-db1f-41ad-9580-70a660434fed 2024-06-15T10:01:54.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:01:54.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:01:54.658 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:01:54.658 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:01:54.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:01:54.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:01:54.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:01:54.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:01:54.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:01:54.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:01:54.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:01:54.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:01:54.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:55.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:01:55.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084290 2024-06-15T10:01:55.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084290 2024-06-15T10:01:55.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290' 2024-06-15T10:01:55.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:55.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:01:55.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-06-15T10:01:55.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-06-15T10:01:55.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-42949672991' 2024-06-15T10:01:55.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:01:55.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:01:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-15T10:01:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-15T10:01:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-42949672991 2-64424509459' 2024-06-15T10:01:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:55.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084290 2024-06-15T10:01:55.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:55.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:01:55.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084290 2024-06-15T10:01:55.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:55.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084290 2024-06-15T10:01:55.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2024-06-15T10:01:55.556 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 120259084290 2024-06-15T10:01:55.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:01:55.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084290 -lt 120259084290 2024-06-15T10:01:55.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:55.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:55.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-06-15T10:01:55.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:01:55.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-06-15T10:01:55.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:55.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-06-15T10:01:55.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-06-15T10:01:55.843 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672991 2024-06-15T10:01:55.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:01:56.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-06-15T10:01:56.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:01:56.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-15T10:01:56.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:01:56.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:01:56.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-15T10:01:56.143 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:01:56.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-15T10:01:56.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-15T10:01:56.145 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509459 2024-06-15T10:01:56.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:56.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509459 2024-06-15T10:01:56.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:01:57.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:01:57.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:01:57.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-06-15T10:01:57.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:01:57.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:57.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:57.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:01:57.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:57.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:57.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:57.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:58.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:01:58.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:58.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:58.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:01:58.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:01:58.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:01:58.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:58.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:58.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:58.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:58.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:58.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:58.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:58.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:01:58.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:58.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:58.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:01:59.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:01:59.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T10:01:59.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:01:59.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:01:59.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:01:59.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:01:59.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:01:59.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:01:59.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:01:59.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:01:59.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:01:59.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:01:59.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:01:59.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:01:59.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:01:59.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:01:59.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:01:59.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:01:59.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:01:59.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:01:59.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:01:59.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:01:59.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:01:59.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:01:59.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:01:59.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:02:00.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:02:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:02:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:02:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:02:00.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:02:00.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:29.093+0000 7fd5e2d6c640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:29.093+0000 7fd5e2d6c640 15 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:29.093+0000 7fd5e2d6c640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:30.105+0000 7fd5e2d6c640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:30.105+0000 7fd5e2d6c640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:30.105+0000 7fd5e2d6c640 15 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-15T10:02:00.059 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:30.105+0000 7fd5e2d6c640 20 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-15T10:02:00.060 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:01:30.105+0000 7fd5e2d6c640 19 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=16/16 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T10:02:00.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-06-15T10:02:00.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:02:00.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:02:00.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:02:00.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:02:00.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:02:00.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:02:00.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:02:00.533 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:02:00.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:02:00.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:02:00.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:02:00.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:02:00.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:02:00.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:02:00.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:02:00.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:02:00.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:02:00.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:02:00.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:02:00.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:02:00.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:02:00.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:00.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:00.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:02:00.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:02:00.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:02:00.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:02:00.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:02:00.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:02:00.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:02:00.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:02:00.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:02:00.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:02:00.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:02:00.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:02:00.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:02:00.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:02:00.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:02:00.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:02:00.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:02:00.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:02:00.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:02:00.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:02:00.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:02:00.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:02:00.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:00.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:00.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:02:00.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:02:00.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:02:00.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:02:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:02:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:00.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:00.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:02:00.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-06-15T10:02:00.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:02:00.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:02:01.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:02:01.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:01.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:01.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:02:01.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:02:01.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:02:01.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:02:01.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:02:01.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:02:01.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:02:01.233 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:02:01.233 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:02:01.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:02:01.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:02:01.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:02:01.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:02:01.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:02:01.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:02:01.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:02:01.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:02:01.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:02:02.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:02:02.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:02.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:02.481 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:02.481 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:02.482 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:02.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:02.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:02:02.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:02:02.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:02:02.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-06-15T10:02:02.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:02:02.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:02.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:02:02.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:02:02.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:02:02.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:02:02.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 020fc962-3200-4485-94c8-c674fd8eb16b' 2024-06-15T10:02:02.528 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:02:02.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:02:02.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaZm1mk5XnIBAAbqNETF8vzMF+F+ihMu5tyg== 2024-06-15T10:02:02.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaZm1mk5XnIBAAbqNETF8vzMF+F+ihMu5tyg=="}' 2024-06-15T10:02:02.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 020fc962-3200-4485-94c8-c674fd8eb16b -i td/osd-scrub-repair/0/new.json 2024-06-15T10:02:02.839 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:02:02.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:02:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCaZm1mk5XnIBAAbqNETF8vzMF+F+ihMu5tyg== --osd-uuid 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:02:02.897 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.900+0000 7fa79aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:02.923 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.928+0000 7fa79aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:02.925 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.928+0000 7fa79aeb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:02.926 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.928+0000 7fa79aeb87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:02:02.926 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.928+0000 7fa79aeb87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:02:02.926 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:02.928+0000 7fa79aeb87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:02:03.023 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:03.028+0000 7fa79aeb87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:02:10.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:02:10.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:02:10.626 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:02:10.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:02:10.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:02:10.900 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:02:10.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:02:10.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:10.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:02:10.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:02:10.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:02:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:10.936+0000 7f96446217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:10.980 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:10.984+0000 7f96446217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:10.981 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:10.984+0000 7f96446217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:11.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:11.134 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:02:11.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:02:11.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:11.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:11.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:11.553 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:11.556+0000 7f96446217c0 -1 Falling back to public interface 2024-06-15T10:02:12.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:12.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:12.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:02:12.388 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:02:12.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:12.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:12.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:13.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:13.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:13.630 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:02:13.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:02:13.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:13.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:13.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:14.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:14.428+0000 7f96446217c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:02:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:14.868 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:02:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:02:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:15.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:16.134 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:02:16.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:02:16.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:16.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:16.370 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2828726740,v1:127.0.0.1:6803/2828726740] [v2:127.0.0.1:6804/2828726740,v1:127.0.0.1:6805/2828726740] exists,up 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:02:16.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:02:16.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:16.374 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:02:16.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:02:16.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05c76eed-749d-43c3-8f03-98f103ad50d1 2024-06-15T10:02:16.378 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 05c76eed-749d-43c3-8f03-98f103ad50d1 2024-06-15T10:02:16.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 05c76eed-749d-43c3-8f03-98f103ad50d1' 2024-06-15T10:02:16.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:02:16.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoZm1m8UlaGBAAWe6sh0/sTTIprF2+lQk08w== 2024-06-15T10:02:16.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoZm1m8UlaGBAAWe6sh0/sTTIprF2+lQk08w=="}' 2024-06-15T10:02:16.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05c76eed-749d-43c3-8f03-98f103ad50d1 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:02:16.923 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:02:16.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:02:16.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCoZm1m8UlaGBAAWe6sh0/sTTIprF2+lQk08w== --osd-uuid 05c76eed-749d-43c3-8f03-98f103ad50d1 2024-06-15T10:02:16.990 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:16.992+0000 7f944eeb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:17.051 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.052+0000 7f944eeb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:17.052 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.056+0000 7f944eeb77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:17.053 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.056+0000 7f944eeb77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:02:17.053 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.056+0000 7f944eeb77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:02:17.053 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.056+0000 7f944eeb77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:02:17.134 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:17.136+0000 7f944eeb77c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:02:25.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:02:25.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:02:25.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:02:25.083 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:02:25.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:02:25.353 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:02:25.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:02:25.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:25.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:02:25.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:02:25.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:02:25.388 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:25.392+0000 7f6db699d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:25.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:25.432+0000 7f6db699d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:25.431 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:25.436+0000 7f6db699d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:25.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:02:25.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:02:25.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:02:25.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:02:25.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:02:25.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:25.648 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:02:25.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:02:25.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:25.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:02:25.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:25.986 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:25.988+0000 7f6db699d7c0 -1 Falling back to public interface 2024-06-15T10:02:26.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:26.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:26.882 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:02:26.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:02:26.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:26.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:02:27.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:28.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:28.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:28.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:02:28.124 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:02:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:02:28.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:28.959 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:28.964+0000 7f6db699d7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:02:29.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:29.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:29.367 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:02:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:02:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:02:29.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:30.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:30.819 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:02:30.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:02:30.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:30.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:02:31.061 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1516218908,v1:127.0.0.1:6811/1516218908] [v2:127.0.0.1:6812/1516218908,v1:127.0.0.1:6813/1516218908] exists,up 05c76eed-749d-43c3-8f03-98f103ad50d1 2024-06-15T10:02:31.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:02:31.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:02:31.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:02:31.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:02:31.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:31.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:02:31.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:02:31.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:02:31.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=293ef95a-2534-4b5c-985e-53aabb2e18bb 2024-06-15T10:02:31.069 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 293ef95a-2534-4b5c-985e-53aabb2e18bb 2024-06-15T10:02:31.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 293ef95a-2534-4b5c-985e-53aabb2e18bb' 2024-06-15T10:02:31.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:02:31.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3Zm1mzUOcBRAAMSPMfJ2EoPaaH/WwIG7bOQ== 2024-06-15T10:02:31.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3Zm1mzUOcBRAAMSPMfJ2EoPaaH/WwIG7bOQ=="}' 2024-06-15T10:02:31.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 293ef95a-2534-4b5c-985e-53aabb2e18bb -i td/osd-scrub-repair/2/new.json 2024-06-15T10:02:31.540 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:02:31.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:02:31.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQC3Zm1mzUOcBRAAMSPMfJ2EoPaaH/WwIG7bOQ== --osd-uuid 293ef95a-2534-4b5c-985e-53aabb2e18bb 2024-06-15T10:02:31.597 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.600+0000 7f3e5676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:31.667 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.672+0000 7f3e5676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:31.669 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.672+0000 7f3e5676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:31.669 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.672+0000 7f3e5676f7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:02:31.669 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.672+0000 7f3e5676f7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:02:31.669 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.672+0000 7f3e5676f7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:02:31.748 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:31.752+0000 7f3e5676f7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:02:39.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:02:39.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:02:39.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:02:39.739 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:02:39.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:02:40.014 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:02:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:02:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:02:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:02:40.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:02:40.049 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:40.052+0000 7fcbbe8057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:40.097 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:40.100+0000 7fcbbe8057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:40.099 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:40.104+0000 7fcbbe8057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:02:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:40.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:02:40.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:40.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:02:40.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:40.674 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:40.676+0000 7fcbbe8057c0 -1 Falling back to public interface 2024-06-15T10:02:41.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:41.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:41.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:02:41.506 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:02:41.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:41.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:02:41.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:42.751 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:02:42.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:02:42.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:42.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:02:42.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:43.640 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:43.644+0000 7fcbbe8057c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:02:43.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:43.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:43.993 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:02:43.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:02:43.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:43.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:02:44.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:45.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:45.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:45.418 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:02:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:02:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:45.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:02:45.659 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2749147644,v1:127.0.0.1:6819/2749147644] [v2:127.0.0.1:6820/2749147644,v1:127.0.0.1:6821/2749147644] exists,up 293ef95a-2534-4b5c-985e-53aabb2e18bb 2024-06-15T10:02:45.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:02:45.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:02:45.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:02:45.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-06-15T10:02:45.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-15T10:02:46.151 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T10:02:46.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:02:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-06-15T10:02:47.924 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T10:02:47.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-06-15T10:02:47.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:02:47.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:02:47.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:02:47.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:02:47.950 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:02:47.950 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:02:47.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:02:47.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:02:47.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:02:48.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:02:48.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:02:48.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:02:48.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:02:48.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:02:48.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:02:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:02:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:02:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:02:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:02:48.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:02:48.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:02:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-06-15T10:02:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-06-15T10:02:48.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-06-15T10:02:48.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:02:48.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:02:48.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-06-15T10:02:48.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-06-15T10:02:48.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2024-06-15T10:02:48.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:02:48.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:02:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-06-15T10:02:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-06-15T10:02:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-64424509442' 2024-06-15T10:02:49.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:02:49.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-06-15T10:02:49.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:02:49.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:02:49.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-06-15T10:02:49.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:02:49.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-06-15T10:02:49.130 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836488 2024-06-15T10:02:49.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-06-15T10:02:49.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:02:49.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836488 2024-06-15T10:02:49.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:02:49.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-06-15T10:02:49.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:02:49.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:02:49.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-06-15T10:02:49.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:02:49.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-06-15T10:02:49.516 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672965 2024-06-15T10:02:49.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-06-15T10:02:49.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:02:49.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672965 2024-06-15T10:02:49.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:02:50.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:02:50.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:02:51.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-06-15T10:02:51.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:02:51.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-06-15T10:02:51.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:02:51.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:02:51.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-06-15T10:02:51.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:02:51.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-06-15T10:02:51.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-06-15T10:02:51.033 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509442 2024-06-15T10:02:51.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:02:51.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509442 2024-06-15T10:02:51.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:02:51.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:02:51.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:02:51.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:02:51.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:02:51.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:02:51.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:02:51.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:02:51.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:02:51.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-06-15T10:02:52.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:02:52.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-15T10:02:52.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T10:02:52.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:02:52.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T10:02:52.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:02:52.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:02:52.302 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:02:52.302 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:02:52.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:02:52.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:02:52.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:02:52.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:02:52.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:02:52.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:02:54.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-15T10:02:55.904 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T10:02:56.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:02:56.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:02:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:02:56.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:56.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:02:56.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:02:56.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:02:56.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:02:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:02:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:02:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:02:56.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:02:56.481 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:56.484+0000 7f9be46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:56.484 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:56.488+0000 7f9be46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:56.486 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:56.488+0000 7f9be46ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:02:56.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:56.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:57.025 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:57.028+0000 7f9be46ba7c0 -1 Falling back to public interface 2024-06-15T10:02:57.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:02:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:02:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:58.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:59.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:02:59.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:02:59.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:02:59.167 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:02:59.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:02:59.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:02:59.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:02:59.936 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:02:59.940+0000 7f9be46ba7c0 -1 osd.0 21 log_to_monitors true 2024-06-15T10:03:00.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:00.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:00.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:03:00.412 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:03:00.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:00.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:00.757 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2683665107,v1:127.0.0.1:6803/2683665107] [v2:127.0.0.1:6804/2683665107,v1:127.0.0.1:6805/2683665107] exists,up 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:03:00.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:03:00.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:03:00.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:03:01.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:03:01.101 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:01.101 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:03:01.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:03:01.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:01.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:03:01.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-15T10:03:01.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-15T10:03:01.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-15T10:03:01.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:01.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:03:01.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-06-15T10:03:01.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-06-15T10:03:01.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672969' 2024-06-15T10:03:01.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:01.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:03:01.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-06-15T10:03:01.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-06-15T10:03:01.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672969 2-64424509446' 2024-06-15T10:03:01.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:01.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-15T10:03:01.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:01.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:03:01.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-15T10:03:01.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-15T10:03:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-15T10:03:01.616 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182402 2024-06-15T10:03:01.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:01.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-06-15T10:03:01.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:02.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:03:02.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:03.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-15T10:03:03.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:03.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-06-15T10:03:03.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:03.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:03:03.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-06-15T10:03:03.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:03.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-06-15T10:03:03.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-06-15T10:03:03.156 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672969 2024-06-15T10:03:03.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:03.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-06-15T10:03:03.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:03.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-06-15T10:03:03.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:03.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:03:03.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-06-15T10:03:03.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:03.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-06-15T10:03:03.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-06-15T10:03:03.441 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509446 2024-06-15T10:03:03.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:03.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-06-15T10:03:03.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:03:03.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:03.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:03:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:04.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:04.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:04.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:04.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:04.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:04.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:03:04.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:04.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:04.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:04.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:04.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:04.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:04.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:03:05.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:03:05.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:03:05.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:03:05.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:03:05.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:03:05.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:03:05.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:03:05.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:05.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:03:05.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:05.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:05.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:05.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:05.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:05.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:05.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:05.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:05.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:05.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:05.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:05.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:03:06.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:03:06.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:03:06.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:03:06.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:03:06.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:03:06.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:06.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:03:06.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:06.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:03:06.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:06.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:06.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:03:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:03:07.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:03:07.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-06-15T10:03:07.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T10:03:07.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:03:07.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:07.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:03:07.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-06-15T10:03:07.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-06-15T10:03:07.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:03:07.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:03:07.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:03:07.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:07.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-06-15T10:03:07.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-06-15T10:03:07.642 INFO:tasks.workunit.client.0.smithi116.stdout:Affected PG 1.0 w/ primary 1 2024-06-15T10:03:07.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-06-15T10:03:07.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:03:07.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:03:07.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:03:07.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-06-15T10:02:46.074898+0000 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:03:07.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:03:07.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-06-15T10:02:46.074898+0000 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-15T10:03:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:03:08.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-06-15T10:03:08.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:03:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-06-15T10:03:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:03:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:03:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-06-15T10:03:08.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:03:08.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:03:08.211 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:03:08.211 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:03:08.211 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:03:08.212 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:08.212 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:08.212 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:08.212 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:03:08.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-06-15T10:03:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:03:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-06-15T10:03:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:03:08.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:08.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:03:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-06-15T10:03:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-06-15T10:03:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-06-15T10:03:08.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:08.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:03:08.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-06-15T10:03:08.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-06-15T10:03:08.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672971' 2024-06-15T10:03:08.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:08.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:03:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-15T10:03:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-15T10:03:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672971 2-64424509448' 2024-06-15T10:03:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:08.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-06-15T10:03:08.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:08.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:03:08.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-06-15T10:03:08.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:08.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-06-15T10:03:08.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-06-15T10:03:08.958 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 107374182404 2024-06-15T10:03:08.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:09.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-06-15T10:03:09.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:09.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:09.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-06-15T10:03:09.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:03:09.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-06-15T10:03:09.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:09.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-06-15T10:03:09.168 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672971 2024-06-15T10:03:09.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-06-15T10:03:09.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:09.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672971 2024-06-15T10:03:09.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:10.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:03:10.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:10.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672971 2024-06-15T10:03:10.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:11.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:03:11.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:11.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-06-15T10:03:11.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:11.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-15T10:03:11.878 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:11.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:03:11.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-15T10:03:11.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:11.881 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509448 2024-06-15T10:03:11.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-15T10:03:11.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-15T10:03:11.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:12.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-06-15T10:03:12.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-06-15T10:03:12.166 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15 10:03:12.175082646+00:00 2024-06-15T10:03:12.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-06-15T10:03:12.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:03:12.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:03:12.168 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:12.168 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:12.168 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:12.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:03:12.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-06-15T10:03:12.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-06-15T10:03:12.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:12.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:12.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:12.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:12.447 INFO:tasks.workunit.client.0.smithi116.stdout:{"deep":true,"must":false,"stamp":"2024-06-01T10:01:32.290618+0000"}0 ) state now: "active+clean" 2024-06-15T10:03:12.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-06-15T10:03:12.448 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:12.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:12.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:12.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-06-15T10:03:12.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:12.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:12.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:12.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:12.908 INFO:tasks.workunit.client.0.smithi116.stdout:1 ) state now: "active+clean" 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-06-15T10:03:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:13.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:13.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:13.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:13.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:13.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:13.368 INFO:tasks.workunit.client.0.smithi116.stdout:2 ) state now: "active+clean" 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-06-15T10:03:13.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:13.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:13.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:13.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:13.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:13.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:13.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stdout:3 ) state now: "active+clean" 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-06-15T10:03:13.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:14.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:14.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:14.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:14.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:14.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:14.290 INFO:tasks.workunit.client.0.smithi116.stdout:4 ) state now: "active+clean" 2024-06-15T10:03:14.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stdout:loop --------> 4 2024-06-15T10:03:14.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-06-15T10:03:14.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:14.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:14.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:14.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:14.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:14.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:14.751 INFO:tasks.workunit.client.0.smithi116.stdout:5 ) state now: "active+clean" 2024-06-15T10:03:14.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-06-15T10:03:14.752 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:14.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:14.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:14.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-06-15T10:03:14.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:15.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:15.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:15.211 INFO:tasks.workunit.client.0.smithi116.stdout:6 ) state now: "active+clean" 2024-06-15T10:03:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-06-15T10:03:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:15.212 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:15.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:15.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-06-15T10:03:15.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:15.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:15.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:15.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:15.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:15.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-06-15T10:03:15.672 INFO:tasks.workunit.client.0.smithi116.stdout:7 ) state now: "active+clean" 2024-06-15T10:03:15.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2024-06-15T10:03:15.672 INFO:tasks.workunit.client.0.smithi116.stdout:"active+clean" 2024-06-15T10:03:15.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:15.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-06-15T10:03:15.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-06-15T10:03:15.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-06-15T10:03:15.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-06-15T10:03:15.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-06-15T10:03:15.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-06-15T10:03:15.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-06-15T10:03:16.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.130 INFO:tasks.workunit.client.0.smithi116.stdout:8 ) state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:16.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.131 INFO:tasks.workunit.client.0.smithi116.stdout:found scrub 2024-06-15T10:03:16.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-06-15T10:03:16.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-06-15T10:03:16.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-06-15T10:03:16.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-06-15T10:03:16.132 INFO:tasks.workunit.client.0.smithi116.stdout:initiate_and_fetch_state ret: 0 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-06-15T10:03:16.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.td/osd-scrub-repair.asok 2024-06-15T10:03:16.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-06-15T10:03:16.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:16.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:16.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:16.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.370 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:16.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:16.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-06-15T10:03:16.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-15T10:03:16.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-15T10:03:16.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:16.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:16.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:16.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.828 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:16.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:16.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:16.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-06-15T10:03:16.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-15T10:03:17.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-15T10:03:17.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:17.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:17.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:17.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:17.283 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:17.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:17.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:17.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-06-15T10:03:17.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-15T10:03:17.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-15T10:03:17.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:17.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:17.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:17.743 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-06-15T10:03:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-15T10:03:18.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-15T10:03:18.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:18.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:18.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-06-15T10:03:18.202 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:03:18.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:03:18.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:03:18.429 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:18.429 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:03:18.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:03:18.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:18.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:03:18.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-06-15T10:03:18.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-06-15T10:03:18.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-06-15T10:03:18.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:18.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:03:18.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-06-15T10:03:18.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-06-15T10:03:18.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672974' 2024-06-15T10:03:18.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:18.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:03:18.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-06-15T10:03:18.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-06-15T10:03:18.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672974 2-64424509451' 2024-06-15T10:03:18.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:18.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-06-15T10:03:18.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:18.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:03:18.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-06-15T10:03:18.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:18.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-06-15T10:03:18.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-06-15T10:03:18.896 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 107374182407 2024-06-15T10:03:18.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:19.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-06-15T10:03:19.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:19.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-06-15T10:03:19.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:19.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:03:19.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-06-15T10:03:19.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:19.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-06-15T10:03:19.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-06-15T10:03:19.190 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672974 2024-06-15T10:03:19.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:19.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-06-15T10:03:19.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:19.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:19.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-06-15T10:03:19.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:03:19.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-06-15T10:03:19.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:19.484 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509451 2024-06-15T10:03:19.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-06-15T10:03:19.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-06-15T10:03:19.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:19.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509451 2024-06-15T10:03:19.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:03:20.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:20.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-06-15T10:03:20.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-06-15T10:03:21.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-06-15T10:03:21.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-06-15T10:03:21.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-06-15T10:03:21.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-06-15T10:03:21.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-06-15T10:03:21.426 INFO:tasks.workunit.client.0.smithi116.stdout:wait-scrub-end state now: "active+clean" 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-06-15T10:03:21.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-06-15T10:03:21.638 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T10:03:21.638 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-06-15T10:03:19.435968+0000 21'1 26:52 [1,0] 1 [1,0] 1 21'1 2024-06-15T10:03:18.936451+0000 21'1 2024-06-15T10:03:18.936451+0000 0 4 periodic scrub scheduled @ 2024-06-16T10:03:18.936451+0000 1 0 2024-06-15T10:03:21.638 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T10:03:21.638 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T10:03:21.638 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T10:03:21.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-15T10:03:21.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T10:03:21.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:03:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T10:03:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:03:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:03:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:21.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:03:21.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:03:21.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:21.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:03:22.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:03:22.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:03:22.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:03:22.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:03:22.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:03:22.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:03:23.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:03:23.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-06-15T10:03:24.790 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:03:24.790 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:03:25.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:03:25.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:03:25.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:03:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:03:25.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:03:25.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:03:25.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:03:25.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:03:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:03:25.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:03:25.095 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:03:25.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:03:25.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:03:25.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:03:25.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:03:25.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:03:25.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:03:25.136 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:25.139+0000 7f981c9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:25.139 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:25.143+0000 7f981c9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:25.141 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:25.143+0000 7f981c9db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:03:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:03:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:03:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:25.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:25.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:25.673 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:25.675+0000 7f981c9db7c0 -1 Falling back to public interface 2024-06-15T10:03:26.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:26.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:26.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:03:26.560 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:26.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:26.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:26.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:27.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:03:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:03:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:28.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:28.404 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:28.407+0000 7f981c9db7c0 -1 osd.0 26 log_to_monitors true 2024-06-15T10:03:29.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:29.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:29.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:03:29.041 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:03:29.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:29.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:29.363 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2590144484,v1:127.0.0.1:6803/2590144484] [v2:127.0.0.1:6804/2590144484,v1:127.0.0.1:6805/2590144484] exists,up 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:03:29.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:03:29.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:03:29.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:03:29.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:03:29.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:03:29.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:03:29.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:03:29.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:03:29.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:03:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:03:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:03:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:03:29.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:29.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:03:29.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T10:03:29.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T10:03:29.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-15T10:03:29.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:29.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:03:30.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-06-15T10:03:30.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-06-15T10:03:30.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977' 2024-06-15T10:03:30.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:30.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:03:30.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-06-15T10:03:30.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-06-15T10:03:30.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977 2-64424509454' 2024-06-15T10:03:30.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:30.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-15T10:03:30.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:30.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:03:30.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-15T10:03:30.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:30.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T10:03:30.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-15T10:03:30.357 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 128849018882 2024-06-15T10:03:30.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:30.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-06-15T10:03:30.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:31.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:03:31.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:31.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T10:03:31.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:31.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-06-15T10:03:31.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:31.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:03:31.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-06-15T10:03:31.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-06-15T10:03:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-06-15T10:03:31.923 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672977 2024-06-15T10:03:31.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:32.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-06-15T10:03:32.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:32.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-06-15T10:03:32.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:32.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:03:32.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-06-15T10:03:32.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:32.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-06-15T10:03:32.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-06-15T10:03:32.206 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509454 2024-06-15T10:03:32.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:32.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509454 2024-06-15T10:03:32.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:03:32.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:32.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:32.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:32.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:33.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:03:33.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:33.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:33.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:03:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:03:33.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:03:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-06-15T10:03:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T10:03:33.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:03:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-06-15T10:03:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T10:03:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:03:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:33.260 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:03:33.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:03:33.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-06-15T10:03:33.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T10:03:33.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:03:33.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:03:34.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:03:34.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:03:36.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:03:36.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:03:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:03:36.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:03:36.226 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:03:36.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-06-15T10:03:36.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:03:36.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:03:36.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:03:36.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:03:36.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:03:36.262 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:36.263+0000 7fe86359c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:36.265 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:36.267+0000 7fe86359c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:36.266 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:36.267+0000 7fe86359c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:36.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:03:36.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:03:36.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:03:36.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:03:36.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:03:36.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:36.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:03:36.468 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:03:36.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:36.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:36.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:36.803+0000 7fe86359c7c0 -1 Falling back to public interface 2024-06-15T10:03:37.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:37.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:37.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:03:37.699 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:37.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:37.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:37.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:38.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:38.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:38.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:03:38.945 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:03:38.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:38.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:39.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:39.775 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:39.779+0000 7fe86359c7c0 -1 osd.0 31 log_to_monitors true 2024-06-15T10:03:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:03:40.185 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:03:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:40.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/2645838077,v1:127.0.0.1:6803/2645838077] [v2:127.0.0.1:6804/2645838077,v1:127.0.0.1:6805/2645838077] exists,up 020fc962-3200-4485-94c8-c674fd8eb16b 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:03:40.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:03:40.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:03:40.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:03:40.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:03:40.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:03:40.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:03:40.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:03:40.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:03:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:03:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:03:40.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:03:40.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:03:40.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:03:40.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:41.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:03:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-15T10:03:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-15T10:03:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-06-15T10:03:41.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:41.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:03:41.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-15T10:03:41.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-15T10:03:41.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672981' 2024-06-15T10:03:41.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:03:41.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:03:41.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-06-15T10:03:41.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-06-15T10:03:41.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672981 2-64424509458' 2024-06-15T10:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-06-15T10:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:41.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:03:41.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-06-15T10:03:41.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-15T10:03:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-06-15T10:03:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 150323855362 2024-06-15T10:03:41.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:42.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-06-15T10:03:42.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:03:43.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:03:43.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:03:43.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-15T10:03:43.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:43.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:43.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-15T10:03:43.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:03:43.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-15T10:03:43.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-15T10:03:43.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-15T10:03:43.297 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672981 2024-06-15T10:03:43.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:03:43.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-15T10:03:43.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:03:43.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-06-15T10:03:43.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:03:43.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:03:43.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-06-15T10:03:43.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:03:43.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-06-15T10:03:43.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-15T10:03:43.578 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509458 2024-06-15T10:03:43.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:03:43.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-15T10:03:43.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:03:43.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:43.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:44.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T10:03:44.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:44.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:44.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:44.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:44.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:44.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:44.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:44.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T10:03:44.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:03:44.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T10:03:44.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:44.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:03:44.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:44.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:44.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:44.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:44.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:44.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:44.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:44.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:44.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:44.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:44.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:44.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:45.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:03:45.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:03:45.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:03:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:03:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:03:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:03:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:03:45.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:45.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:45.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:45.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:03:45.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:45.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:45.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T10:03:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-06-15T10:03:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:03:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:03:46.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:03:46.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:03:46.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:03:46.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:03:46.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:03:46.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:03:46.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:03:46.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:03:46.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:03:46.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:03:46.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:03:46.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:03:46.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:03:46.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:03:46.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:03:46.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:03:46.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:03:46.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:03:47.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T10:03:47.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:03:47.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:03:47.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:03:47.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T10:03:47.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:03:47.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:03:47.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:03:47.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-06-15T10:03:47.383 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:03:18.931+0000 7f6d9d12c640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-06-15T10:03:47.383 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:03:18.931+0000 7f6d9d12c640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-06-15T10:03:47.384 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:03:18.931+0000 7f6d9d12c640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-06-15T10:03:47.384 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:03:18.931+0000 7f6d9d12c640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-06-15T10:03:47.384 INFO:tasks.workunit.client.0.smithi116.stdout:2024-06-15T10:03:18.931+0000 7f6d9d12c640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:03:47.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:03:47.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:03:47.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:03:47.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:03:47.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:03:47.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:03:47.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:03:47.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:03:47.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:03:47.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:03:47.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:03:47.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:03:47.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:03:47.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:03:47.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:03:47.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:03:47.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:47.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:47.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:03:47.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:03:47.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:03:47.721 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:03:47.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:03:47.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:03:47.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:03:47.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:03:47.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:03:47.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:03:47.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:03:47.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:03:47.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:03:47.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:03:47.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:03:47.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:03:47.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:03:47.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:03:47.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:03:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:03:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:03:47.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:03:47.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:47.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:47.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:03:47.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:03:47.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:03:47.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:03:47.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:03:47.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:47.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:47.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:03:47.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:03:47.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:03:47.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:03:47.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:03:47.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:03:48.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:03:48.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:03:48.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:03:48.151 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:03:48.151 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:48.151 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:48.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:48.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:03:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:48.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:48.390 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:03:48.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:03:48.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:03:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:03:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:03:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:03:48.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:03:48.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:03:48.510 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:03:48.510 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:03:48.510 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:03:48.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:03:48.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:03:48.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:03:49.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:03:49.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:03:49.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:03:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:03:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:49.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:49.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:03:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:03:49.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:03:49.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:03:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:03:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:03:49.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74ed2c8e-3cef-4313-991b-9112eb1b3c29 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 74ed2c8e-3cef-4313-991b-9112eb1b3c29' 2024-06-15T10:03:49.597 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 74ed2c8e-3cef-4313-991b-9112eb1b3c29 2024-06-15T10:03:49.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:03:49.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFZ21mZfAQJRAAXLlKIljcJN7naN4jNHMIIQ== 2024-06-15T10:03:49.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFZ21mZfAQJRAAXLlKIljcJN7naN4jNHMIIQ=="}' 2024-06-15T10:03:49.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74ed2c8e-3cef-4313-991b-9112eb1b3c29 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:03:49.905 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:03:49.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:03:49.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAFZ21mZfAQJRAAXLlKIljcJN7naN4jNHMIIQ== --osd-uuid 74ed2c8e-3cef-4313-991b-9112eb1b3c29 2024-06-15T10:03:49.959 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.963+0000 7f1e5964b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:49.990 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.991+0000 7f1e5964b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:49.991 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.995+0000 7f1e5964b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:49.991 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.995+0000 7f1e5964b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:03:49.991 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.995+0000 7f1e5964b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:03:49.992 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:49.995+0000 7f1e5964b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:03:50.098 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:50.099+0000 7f1e5964b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:03:57.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:03:57.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:03:57.818 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:03:57.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:03:57.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:03:58.094 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:03:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:03:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:03:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:03:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:03:58.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:03:58.130 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:58.131+0000 7f2da2bd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:58.171 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:58.175+0000 7f2da2bd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:58.172 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:58.175+0000 7f2da2bd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:03:58.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:03:58.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:03:58.310 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:03:58.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:58.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:58.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:03:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:03:58.759+0000 7f2da2bd37c0 -1 Falling back to public interface 2024-06-15T10:03:59.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:03:59.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:03:59.573 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:03:59.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:03:59.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:03:59.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:03:59.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:00.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:00.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:00.813 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:04:00.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:04:00.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:00.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:01.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:01.598 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:01.603+0000 7f2da2bd37c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:04:02.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:02.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:02.047 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:04:02.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:04:02.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:02.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:02.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:03.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:03.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:03.441 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:04:03.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:04:03.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:03.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:03.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:04.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:04.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:04.768 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:04:04.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:04:04.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:04.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:05.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:06.011 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:04:06.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:04:06.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:06.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:06.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:07.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:07.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:07.339 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:04:07.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:04:07.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:07.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:07.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:08.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:08.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:08.694 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:04:08.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:04:08.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:08.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:08.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:09.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:09.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:09.935 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:04:09.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:04:09.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:09.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:10.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:11.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:11.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:11.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:04:11.177 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:04:11.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:11.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:11.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:12.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:12.417 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:04:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:04:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:12.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:12.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:13.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:13.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:13.660 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:04:13.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:04:13.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:13.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:13.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:14.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:14.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:14.905 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:04:14.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:04:14.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:14.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:15.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:16.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:16.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:16.236 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:04:16.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:04:16.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:16.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:16.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:17.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:17.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:17.611 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:04:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:04:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:17.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:18.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:18.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:18.859 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:04:18.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:04:18.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:18.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:19.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:20.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:20.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:20.385 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:04:20.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:04:20.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:20.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:20.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:21.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:21.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:21.976 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:04:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:04:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:22.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:23.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:23.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:23.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:04:23.582 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:04:23.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:23.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:23.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:24.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:24.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:24.884 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:04:24.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:04:24.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:24.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:25.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:26.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:26.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:26.621 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:04:26.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:04:26.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:26.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:27.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:28.220 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:04:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:04:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:28.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:28.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:29.912 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:04:29.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:04:29.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:29.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:30.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:31.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:31.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:31.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:04:31.151 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:04:31.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:31.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:31.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:32.411 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:04:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:04:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:32.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:04:32.650 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/755622260,v1:127.0.0.1:6803/755622260] [v2:127.0.0.1:6804/755622260,v1:127.0.0.1:6805/755622260] exists,up 74ed2c8e-3cef-4313-991b-9112eb1b3c29 2024-06-15T10:04:32.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:04:32.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:04:32.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:04:32.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:04:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:04:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:04:32.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:04:32.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:04:32.657 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:04:32.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2e8611d5-ee09-4e14-871d-8eabea89ef45' 2024-06-15T10:04:32.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:04:32.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwZ21mkPfTKBAAzX8G569WbupR55oATf9wxA== 2024-06-15T10:04:32.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwZ21mkPfTKBAAzX8G569WbupR55oATf9wxA=="}' 2024-06-15T10:04:32.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e8611d5-ee09-4e14-871d-8eabea89ef45 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:04:32.987 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:04:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:04:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAwZ21mkPfTKBAAzX8G569WbupR55oATf9wxA== --osd-uuid 2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:04:33.043 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.051+0000 7f98041127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:33.114 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.123+0000 7f98041127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.123+0000 7f98041127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.123+0000 7f98041127c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:04:33.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.123+0000 7f98041127c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:04:33.117 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.123+0000 7f98041127c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:04:33.198 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:33.203+0000 7f98041127c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:04:41.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:04:41.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:04:41.471 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:04:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:04:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:04:41.753 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:04:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:04:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:04:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:04:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:04:41.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:04:41.789 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:41.795+0000 7f9241ce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:41.843+0000 7f9241ce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:41.847+0000 7f9241ce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:04:41.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:04:41.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:04:41.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:04:41.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:41.980 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:04:41.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:04:41.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:41.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:42.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:42.405 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:42.411+0000 7f9241ce07c0 -1 Falling back to public interface 2024-06-15T10:04:43.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:43.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:43.220 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:04:43.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:04:43.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:43.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:43.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:44.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:04:44.460 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:04:44.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:44.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:44.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:04:45.419+0000 7f9241ce07c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:04:45.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:45.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:45.762 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:04:45.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:04:45.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:45.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:46.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:47.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:47.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:47.350 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:04:47.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:04:47.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:47.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:47.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:48.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:48.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:48.793 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:04:48.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:04:48.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:48.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:49.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:50.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:50.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:50.154 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:04:50.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:04:50.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:50.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:50.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:51.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:51.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:51.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:04:51.492 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:04:51.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:51.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:51.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:52.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:52.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:52.735 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:04:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:04:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:52.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:53.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:54.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:54.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:54.059 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:04:54.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:04:54.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:54.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:54.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:55.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:55.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:55.396 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:04:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:04:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:55.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:56.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:56.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:56.712 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:04:56.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:04:56.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:56.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:57.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:58.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:58.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:58.206 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:04:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:04:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:58.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:04:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:04:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:04:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:04:59.542 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:04:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:04:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:04:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:00.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:05:01.002 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:05:01.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:01.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:01.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:02.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:02.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:02.404 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:05:02.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:05:02.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:02.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:02.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:03.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:03.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:03.757 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:05:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:05:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:04.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:05.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:05.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:05.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:05:05.052 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:05:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:05.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:06.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:06.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:06.296 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:05:06.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:05:06.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:06.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:06.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:07.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:07.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:07.943 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:05:07.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:05:07.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:07.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:08.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:09.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:09.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:09.679 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:05:09.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:05:09.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:09.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:10.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:11.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:11.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:11.348 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:05:11.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:05:11.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:11.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:11.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:12.665 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:05:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:05:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:12.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:13.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:14.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:14.332 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:05:14.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:05:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:14.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:14.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:15.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:15.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:15.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:05:15.938 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:05:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:16.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:17.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:17.564 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:05:17.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:05:17.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:17.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:05:17.804 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2017794839,v1:127.0.0.1:6811/2017794839] [v2:127.0.0.1:6812/2017794839,v1:127.0.0.1:6813/2017794839] exists,up 2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:05:17.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:05:17.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:05:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca 2024-06-15T10:05:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca' 2024-06-15T10:05:17.813 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca 2024-06-15T10:05:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdZ21ms84QMhAAvTbXHtG3UZmliTgTP2Zfug== 2024-06-15T10:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdZ21ms84QMhAAvTbXHtG3UZmliTgTP2Zfug=="}' 2024-06-15T10:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca -i td/osd-scrub-repair/2/new.json 2024-06-15T10:05:18.180 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:05:18.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:05:18.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBdZ21ms84QMhAAvTbXHtG3UZmliTgTP2Zfug== --osd-uuid 9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca 2024-06-15T10:05:18.240 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.246+0000 7f34fcbec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:18.315 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.322+0000 7f34fcbec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:18.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.322+0000 7f34fcbec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:18.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.322+0000 7f34fcbec7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:05:18.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.322+0000 7f34fcbec7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:05:18.318 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.322+0000 7f34fcbec7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:05:18.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:18.430+0000 7f34fcbec7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:05:27.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:05:27.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:05:27.669 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:05:27.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:05:27.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:05:27.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:05:27.944 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:05:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:05:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:05:27.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:05:27.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:05:27.979 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:27.986+0000 7f891a9237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:28.036 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:28.042+0000 7f891a9237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:28.038 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:28.046+0000 7f891a9237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:05:28.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:05:28.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:05:28.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:05:28.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:05:28.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:05:28.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:28.223 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:05:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:05:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:28.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:28.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:28.604 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:28.610+0000 7f891a9237c0 -1 Falling back to public interface 2024-06-15T10:05:29.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:29.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:29.452 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:05:29.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:05:29.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:29.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:29.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:30.695 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:05:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:05:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:05:31.006+0000 7f891a9237c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:05:31.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:32.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:32.055 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:05:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:05:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:32.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:32.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:33.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:33.573 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:05:33.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:05:33.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:33.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:33.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:34.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:34.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:34.958 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:05:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:05:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:35.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:36.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:36.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:36.297 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:05:36.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:05:36.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:36.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:36.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:37.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:37.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:37.542 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:05:37.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:05:37.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:37.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:37.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:38.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:38.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:38.880 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:05:38.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:05:38.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:38.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:39.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:40.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:40.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:40.382 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:05:40.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:05:40.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:40.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:40.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:41.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:41.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:41.777 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:05:41.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:05:41.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:41.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:42.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:43.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:43.221 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:05:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:05:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:43.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:44.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:44.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:44.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:05:44.463 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:05:44.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:44.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:44.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:45.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:45.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:45.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:05:45.782 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:05:45.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:45.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:46.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:47.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:47.305 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:05:47.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:05:47.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:47.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:47.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:48.546 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:05:48.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:05:48.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:48.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:49.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:50.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:50.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:50.051 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:05:50.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:05:50.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:50.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:50.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:51.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:05:51.332 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:05:51.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:51.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:51.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:52.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:52.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:52.957 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:05:52.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:05:52.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:52.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:53.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:54.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:54.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:54.317 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:05:54.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:05:54.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:54.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:54.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:55.887 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:05:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:05:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:56.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:57.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:57.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:57.259 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:05:57.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:05:57.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:57.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:57.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:05:58.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:05:58.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:05:58.503 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:05:58.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:05:58.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:05:58.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:05:59.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:00.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:00.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:00.156 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:06:00.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:06:00.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:00.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:06:00.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:01.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:01.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:01.508 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:06:01.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:06:01.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:01.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:06:02.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:03.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:03.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:03.040 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:06:03.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:06:03.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:03.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:06:03.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:04.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:04.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:04.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:06:04.740 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:06:04.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:04.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:06:05.010 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1496432897,v1:127.0.0.1:6819/1496432897] [v2:127.0.0.1:6820/1496432897,v1:127.0.0.1:6821/1496432897] exists,up 9fa89fc5-fb55-4aa8-abe5-7aee5145c8ca 2024-06-15T10:06:05.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:06:05.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:06:05.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:06:05.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-15T10:06:05.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:06:05.397 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:06:05.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:06:05.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:06:05.822 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:06:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:06:06.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:06:10.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-15T10:06:10.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:06:10.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:06:10.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:06:10.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:06:10.274 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:06:10.274 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:06:10.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:06:10.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:06:10.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:06:10.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:06:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:06:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:06:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:06:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:10.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:06:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-15T10:06:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-15T10:06:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-15T10:06:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:10.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:06:10.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411342 2024-06-15T10:06:10.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411342 2024-06-15T10:06:10.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-77309411342' 2024-06-15T10:06:10.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724551 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724551 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-77309411342 2-167503724551' 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-15T10:06:11.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:11.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:06:11.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-15T10:06:11.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:11.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-15T10:06:11.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-15T10:06:11.140 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836507 2024-06-15T10:06:11.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:11.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-15T10:06:11.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:06:12.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:06:12.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:12.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-15T10:06:12.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:12.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411342 2024-06-15T10:06:12.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:12.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:06:12.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411342 2024-06-15T10:06:12.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:12.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411342 2024-06-15T10:06:12.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411342' 2024-06-15T10:06:12.650 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 77309411342 2024-06-15T10:06:12.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411342 -lt 77309411342 2024-06-15T10:06:12.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:12.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724551 2024-06-15T10:06:12.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:12.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:06:12.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724551 2024-06-15T10:06:12.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:12.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724551 2024-06-15T10:06:12.925 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 167503724551 2024-06-15T10:06:12.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724551' 2024-06-15T10:06:12.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:06:13.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724551 -lt 167503724551 2024-06-15T10:06:13.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:06:13.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:13.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:13.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:06:13.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:13.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:06:13.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:13.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:13.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:13.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:06:13.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-15T10:06:14.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:06:14.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:06:15.949 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:06:15.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:06:16.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T10:06:16.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:06:16.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:06:16.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:06:16.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:06:16.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:06:16.970 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:06:16.970 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:06:16.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:06:16.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:06:16.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:06:17.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:06:17.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:06:17.336 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:17.336 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:06:17.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:06:17.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:17.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:06:17.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-06-15T10:06:17.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-06-15T10:06:17.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-06-15T10:06:17.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:17.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:06:17.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411344 2024-06-15T10:06:17.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411344 2024-06-15T10:06:17.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-77309411344' 2024-06-15T10:06:17.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:17.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:06:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724553 2024-06-15T10:06:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724553 2024-06-15T10:06:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-77309411344 2-167503724553' 2024-06-15T10:06:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:17.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-06-15T10:06:17.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:17.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:06:17.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-06-15T10:06:17.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:17.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-06-15T10:06:17.792 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836510 2024-06-15T10:06:17.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-06-15T10:06:17.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:18.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-06-15T10:06:18.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:18.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411344 2024-06-15T10:06:18.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:18.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:06:18.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411344 2024-06-15T10:06:18.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:18.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411344 2024-06-15T10:06:18.239 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 77309411344 2024-06-15T10:06:18.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411344' 2024-06-15T10:06:18.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411344 -lt 77309411344 2024-06-15T10:06:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:18.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724553 2024-06-15T10:06:18.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:18.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:06:18.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724553 2024-06-15T10:06:18.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:18.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724553 2024-06-15T10:06:18.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724553' 2024-06-15T10:06:18.524 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 167503724553 2024-06-15T10:06:18.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:06:18.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724553 -lt 167503724553 2024-06-15T10:06:18.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:06:18.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:18.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:19.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:06:19.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:19.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:19.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:19.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:19.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:19.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:19.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:19.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:06:19.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:19.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:19.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:19.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:06:19.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:06:19.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:06:19.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:06:19.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:06:19.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:06:19.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:06:19.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:19.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:19.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:06:19.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:19.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:19.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:20.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-15T10:06:20.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:06:20.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-15T10:06:20.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-15T10:06:20.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:06:20.528 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-15T10:06:20.528 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:06:20.528 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:06:20.528 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:06:20.528 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:06:20.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-15T10:06:20.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:06:20.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:06:21.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:06:21.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:06:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:06:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:06:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:06:22.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:06:22.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:06:22.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:06:22.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:06:22.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:06:24.418 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:06:24.419 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-15T10:06:24.419 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:06:24.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:06:24.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:06:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:06:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:24.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:06:24.964 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:06:24.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:06:24.998 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:25.005+0000 7f4b6cab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:25.005+0000 7f4b6cab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:25.009+0000 7f4b6cab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:25.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:06:25.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:25.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:25.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:25.536 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:25.541+0000 7f4b6cab47c0 -1 Falling back to public interface 2024-06-15T10:06:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:06:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:26.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:27.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:27.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:27.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:06:27.691 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:06:27.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:27.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:27.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:28.633 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:28.637+0000 7f4b6cab47c0 -1 osd.1 59 log_to_monitors true 2024-06-15T10:06:28.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:28.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:28.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:06:28.954 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:06:28.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:28.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:29.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:06:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:06:30.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:30.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 61 last_clean_interval [18,59) [v2:127.0.0.1:6810/2751112173,v1:127.0.0.1:6811/2751112173] [v2:127.0.0.1:6812/2751112173,v1:127.0.0.1:6813/2751112173] exists,up 2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:06:30.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:06:30.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:06:30.803 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:06:30.803 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:06:30.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:06:30.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:06:30.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:06:30.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:06:30.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:06:30.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:06:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:06:30.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:06:30.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:06:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:06:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:06:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:06:31.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:31.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:06:31.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-06-15T10:06:31.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-06-15T10:06:31.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-06-15T10:06:31.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:31.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:06:31.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-06-15T10:06:31.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-06-15T10:06:31.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-287762808834' 2024-06-15T10:06:31.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:31.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724557 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724557 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-287762808834 2-167503724557' 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-06-15T10:06:31.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:31.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:06:31.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:31.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-06-15T10:06:31.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-06-15T10:06:31.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-15T10:06:31.746 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836513 2024-06-15T10:06:31.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:32.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-06-15T10:06:32.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:32.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-06-15T10:06:32.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:32.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:06:32.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-06-15T10:06:32.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:32.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-06-15T10:06:32.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-06-15T10:06:32.037 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 287762808834 2024-06-15T10:06:32.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:32.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-06-15T10:06:32.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:32.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724557 2024-06-15T10:06:32.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:32.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:06:32.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724557 2024-06-15T10:06:32.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:32.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724557 2024-06-15T10:06:32.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724557' 2024-06-15T10:06:32.334 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 167503724557 2024-06-15T10:06:32.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:06:32.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724557 -lt 167503724557 2024-06-15T10:06:32.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:06:32.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:32.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:06:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:33.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:06:33.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:33.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:33.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:33.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:06:33.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:06:33.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:06:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-15T10:06:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:06:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:06:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:06:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:06:33.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-15T10:06:33.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-15T10:06:33.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:06:33.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:06:33.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:06:33.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-15T10:06:15.665220+0000 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:06:15.665220+0000 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:06:34.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:06:34.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:06:34.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:06:34.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:06:34.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:06:34.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:06:15.665220+0000 '>' 2024-06-15T10:06:15.665220+0000 2024-06-15T10:06:34.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:06:35.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:06:35.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:06:35.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:06:35.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:06:35.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:06:35.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:06:35.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:06:35.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:06:34.289466+0000 '>' 2024-06-15T10:06:15.665220+0000 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:06:35.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:06:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:06:35.550 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:06:35.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:06:35.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:06:35.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:06:35.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:06:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:06:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:06:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:06:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:35.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:06:36.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-06-15T10:06:36.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-06-15T10:06:36.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-06-15T10:06:36.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:36.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:06:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808836 2024-06-15T10:06:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808836 2024-06-15T10:06:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-287762808836' 2024-06-15T10:06:36.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:36.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:06:36.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724559 2024-06-15T10:06:36.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724559 2024-06-15T10:06:36.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-287762808836 2-167503724559' 2024-06-15T10:06:36.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:36.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-06-15T10:06:36.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:36.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:06:36.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-06-15T10:06:36.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:36.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-06-15T10:06:36.395 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836515 2024-06-15T10:06:36.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-06-15T10:06:36.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-06-15T10:06:36.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:06:37.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:06:37.619 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:37.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-06-15T10:06:37.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:06:38.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:06:38.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:39.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836515 2024-06-15T10:06:39.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:39.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:39.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808836 2024-06-15T10:06:39.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:06:39.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:39.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808836 2024-06-15T10:06:39.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808836 2024-06-15T10:06:39.108 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 287762808836 2024-06-15T10:06:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808836' 2024-06-15T10:06:39.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808836 -lt 287762808836 2024-06-15T10:06:39.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:39.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724559 2024-06-15T10:06:39.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:06:39.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724559 2024-06-15T10:06:39.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:39.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724559 2024-06-15T10:06:39.595 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 167503724559 2024-06-15T10:06:39.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724559' 2024-06-15T10:06:39.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:06:39.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724559 -lt 167503724559 2024-06-15T10:06:39.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:06:39.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:39.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:40.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:06:40.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:40.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:40.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:06:40.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:40.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:40.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:40.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:06:40.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:06:40.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:06:40.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:06:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:06:40.580 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:06:40.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-15T10:06:40.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-15T10:06:40.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:06:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:06:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:06:41.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:06:44.137 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:06:44.138 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-15T10:06:44.138 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:06:44.138 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:06:44.138 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:06:44.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:06:44.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:06:44.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:06:44.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:06:44.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:06:44.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:06:44.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:06:44.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:06:44.438 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:06:44.438 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:44.438 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:06:44.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:06:44.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:06:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:06:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:06:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:06:44.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:06:44.477 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:44.481+0000 7fb60a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:44.479 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:44.485+0000 7fb60a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:44.481 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:44.485+0000 7fb60a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:06:44.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:44.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:44.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:45.017 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:45.021+0000 7fb60a4e27c0 -1 Falling back to public interface 2024-06-15T10:06:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:06:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:46.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:47.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:06:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:06:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:47.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:47.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:06:47.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:47.841+0000 7fb60a4e27c0 -1 osd.1 74 log_to_monitors true 2024-06-15T10:06:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:06:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:06:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:06:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:06:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:06:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:06:48.782 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 82 up_thru 67 down_at 76 last_clean_interval [67,74) [v2:127.0.0.1:6810/3142182812,v1:127.0.0.1:6811/3142182812] [v2:127.0.0.1:6812/3142182812,v1:127.0.0.1:6813/3142182812] exists,up 2e8611d5-ee09-4e14-871d-8eabea89ef45 2024-06-15T10:06:48.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:06:48.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:06:48.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:06:48.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:06:48.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:06:48.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:06:48.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:06:48.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:06:48.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:06:48.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:06:48.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:06:48.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:06:48.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:06:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:06:49.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:06:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:06:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:06:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:06:49.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:49.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:06:49.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-06-15T10:06:49.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-06-15T10:06:49.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-06-15T10:06:49.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:49.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:06:49.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-06-15T10:06:49.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-06-15T10:06:49.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-352187318274' 2024-06-15T10:06:49.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:06:49.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:06:50.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724563 2024-06-15T10:06:50.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724563 2024-06-15T10:06:50.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-352187318274 2-167503724563' 2024-06-15T10:06:50.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:50.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-06-15T10:06:50.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:50.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:06:50.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-06-15T10:06:50.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:50.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-06-15T10:06:50.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-06-15T10:06:50.120 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836519 2024-06-15T10:06:50.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:06:50.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-06-15T10:06:50.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:50.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-06-15T10:06:50.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:50.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:06:50.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-06-15T10:06:50.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-06-15T10:06:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-06-15T10:06:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 352187318274 2024-06-15T10:06:50.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:51.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-06-15T10:06:51.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:06:52.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:06:52.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:06:52.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-06-15T10:06:52.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:06:52.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:06:52.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-167503724563 2024-06-15T10:06:52.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:06:52.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-167503724563 2024-06-15T10:06:52.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:06:52.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724563 2024-06-15T10:06:52.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 167503724563' 2024-06-15T10:06:52.416 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 167503724563 2024-06-15T10:06:52.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:06:52.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724563 -lt 167503724563 2024-06-15T10:06:52.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:06:52.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:52.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:52.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:06:52.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:52.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:52.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:52.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:52.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:52.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:52.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:53.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T10:06:53.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:53.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:53.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 5 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:06:53.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:06:53.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:06:53.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:53.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:53.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:53.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:53.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:53.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:53.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:53.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:06:53.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:53.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:53.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:53.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:06:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 0 2024-06-15T10:06:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:06:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:06:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:06:53.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:06:54.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:06:54.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:54.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:54.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:54.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:54.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:54.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:54.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:54.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:06:54.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:54.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:54.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:06:54.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:06:54.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:06:54.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:06:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:06:54.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:06:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:06:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:06:54.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:06:55.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:06:55.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:55.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:55.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:55.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:55.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:55.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:55.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:55.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:06:55.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:55.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:55.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:06:55.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:06:55.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:06:55.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:06:55.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:06:55.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:06:55.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:06:55.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:06:55.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:06:56.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:06:56.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:06:56.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:06:56.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:06:56.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:06:56.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:06:56.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:06:56.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:06:56.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:06:56.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:06:56.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:06:56.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:06:56.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:06:56.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:06:56.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:06:56.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:06:56.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:06:56.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:06:56.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:06:56.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:06:56.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:06:56.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:06:56.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:06:56.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:06:56.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:06:56.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:06:57.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:06:57.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:06:57.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:06:57.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:06:57.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:06:57.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:06:57.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:06:57.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:06:57.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:06:57.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:06:57.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:06:57.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:06:57.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:06:57.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:06:57.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:06:57.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:57.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:57.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:06:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:06:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:06:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:06:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:06:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:06:57.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:06:57.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:06:57.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:06:57.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:06:57.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:06:57.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:06:57.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:06:57.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:06:57.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:06:57.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:06:57.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:06:57.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:06:57.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:06:57.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:06:57.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:06:57.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:57.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:57.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:06:57.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:06:57.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:06:57.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:06:57.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:06:57.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:57.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:57.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:06:57.311 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-06-15T10:06:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:06:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:06:57.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:57.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:57.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:06:57.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:06:57.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:06:57.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:06:57.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:06:57.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:06:57.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:06:57.989 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:06:57.989 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:06:57.989 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:06:57.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:06:58.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:06:58.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:06:58.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:06:58.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:58.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:06:58.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:06:58.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:06:58.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-15T10:06:58.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:06:58.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:06:58.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:06:58.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:06:58.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:06:58.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:59.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:59.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:06:59.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:06:59.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:06:59.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:06:59.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:06:59.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:06:59.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:06:59.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59cef25f-bfd0-4e2e-a8c8-52286a28b809 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 59cef25f-bfd0-4e2e-a8c8-52286a28b809' 2024-06-15T10:06:59.152 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 59cef25f-bfd0-4e2e-a8c8-52286a28b809 2024-06-15T10:06:59.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:06:59.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDZ21mGsCWChAAPm3b930DUe/7q9YPWgdO1g== 2024-06-15T10:06:59.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDZ21mGsCWChAAPm3b930DUe/7q9YPWgdO1g=="}' 2024-06-15T10:06:59.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59cef25f-bfd0-4e2e-a8c8-52286a28b809 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:06:59.480 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:06:59.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:06:59.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDDZ21mGsCWChAAPm3b930DUe/7q9YPWgdO1g== --osd-uuid 59cef25f-bfd0-4e2e-a8c8-52286a28b809 2024-06-15T10:06:59.535 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.541+0000 7f2e304ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:59.565 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.569+0000 7f2e304ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:59.566 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.573+0000 7f2e304ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:06:59.567 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.573+0000 7f2e304ae7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:06:59.567 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.573+0000 7f2e304ae7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:06:59.567 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.573+0000 7f2e304ae7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:06:59.664 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:06:59.669+0000 7f2e304ae7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:07:07.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:07:07.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:07:07.568 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:07:07.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:07:07.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:07:07.833 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:07:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:07:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:07:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:07:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:07:07.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:07:07.868 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:07.873+0000 7f556d3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:07.921 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:07.925+0000 7f556d3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:07.923 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:07.929+0000 7f556d3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:08.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:07:08.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:07:08.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:07:08.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:07:08.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:07:08.063 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:07:08.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:08.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:07:08.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:08.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:08.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:08.492 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:08.497+0000 7f556d3b57c0 -1 Falling back to public interface 2024-06-15T10:07:09.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:09.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:09.320 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:07:09.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:07:09.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:09.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:09.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:10.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:10.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:10.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:07:10.555 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:07:10.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:10.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:10.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:11.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:11.405+0000 7f556d3b57c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:07:11.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:11.793 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:07:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:07:11.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:11.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:12.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:13.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:13.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:13.221 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:07:13.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:07:13.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:13.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:13.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:14.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:14.490 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:07:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:07:14.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:14.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:14.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:15.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:15.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:15.731 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:07:15.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:07:15.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:15.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:15.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:16.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:16.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:16.973 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:07:16.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:07:16.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:16.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:17.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:18.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:18.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:18.339 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:07:18.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:07:18.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:18.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:18.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:19.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:19.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:19.583 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:07:19.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:07:19.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:19.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:19.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:20.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:20.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:20.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:07:20.826 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:07:20.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:20.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:21.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:22.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:22.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:22.123 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:07:22.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:07:22.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:22.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:22.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:23.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:23.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:23.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:07:23.458 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:07:23.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:23.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:23.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:24.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:24.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:24.962 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:07:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:07:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:24.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:25.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:26.423 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:07:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:07:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:26.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:27.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:27.794 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:07:27.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:07:27.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:27.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:28.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:29.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:29.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:29.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:07:29.348 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:07:29.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:29.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:29.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:30.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:30.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:30.774 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:07:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:07:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:30.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:31.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:32.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:32.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:32.245 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:07:32.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:07:32.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:32.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:32.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:33.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:33.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:33.516 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:07:33.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:07:33.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:33.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:34.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:35.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:35.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:35.348 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:07:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:07:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:35.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:36.972 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:07:36.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:07:36.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:36.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:37.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:38.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:38.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:38.550 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:07:38.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:07:38.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:38.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:39.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:40.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:40.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:40.260 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:07:40.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:07:40.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:40.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:40.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:41.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:41.670 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:07:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:07:41.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:41.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:07:41.912 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3732927642,v1:127.0.0.1:6803/3732927642] [v2:127.0.0.1:6804/3732927642,v1:127.0.0.1:6805/3732927642] exists,up 59cef25f-bfd0-4e2e-a8c8-52286a28b809 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:07:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:07:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:07:41.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:07:41.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:07:41.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:07:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:07:41.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:07:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:07:41.919 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:07:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 660ce8c9-9eea-48f4-948b-b062306b7a64' 2024-06-15T10:07:41.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:07:41.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtZ21muHqoOBAA2jlUNSOTkcIkV/KvrMXGuQ== 2024-06-15T10:07:41.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtZ21muHqoOBAA2jlUNSOTkcIkV/KvrMXGuQ=="}' 2024-06-15T10:07:41.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 660ce8c9-9eea-48f4-948b-b062306b7a64 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:07:42.260 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:07:42.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:07:42.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDtZ21muHqoOBAA2jlUNSOTkcIkV/KvrMXGuQ== --osd-uuid 660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:07:42.314 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.320+0000 7f00d4f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:42.388 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.392+0000 7f00d4f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:42.389 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.392+0000 7f00d4f717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:42.390 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.396+0000 7f00d4f717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:07:42.390 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.396+0000 7f00d4f717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:07:42.390 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.396+0000 7f00d4f717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:07:42.475 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:42.480+0000 7f00d4f717c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:07:50.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:07:50.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:07:50.608 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:07:50.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:07:50.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:07:50.882 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:07:50.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:07:50.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:07:50.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:07:50.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:07:50.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:07:50.920 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:50.924+0000 7f39c45c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:50.960 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:50.964+0000 7f39c45c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:50.968+0000 7f39c45c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:07:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:51.114 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:07:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:07:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:51.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:51.519 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:51.524+0000 7f39c45c47c0 -1 Falling back to public interface 2024-06-15T10:07:52.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:52.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:52.489 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:07:52.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:07:52.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:52.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:52.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:53.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:53.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:53.730 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:07:53.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:07:53.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:53.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:53.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:54.508 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:07:54.512+0000 7f39c45c47c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:07:54.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:54.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:54.971 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:07:54.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:07:54.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:54.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:56.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:56.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:56.497 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:07:56.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:07:56.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:56.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:56.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:57.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:57.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:57.968 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:07:57.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:07:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:57.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:58.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:07:59.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:07:59.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:07:59.366 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:07:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:07:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:07:59.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:07:59.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:00.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:00.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:00.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:08:00.911 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:08:00.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:00.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:01.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:02.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:02.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:02.151 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:08:02.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:08:02.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:02.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:02.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:03.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:03.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:03.635 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:08:03.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:08:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:04.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:05.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:05.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:05.018 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:08:05.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:08:05.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:05.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:05.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:08:06.257 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:08:06.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:06.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:06.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:07.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:07.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:07.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:08:07.585 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:08:07.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:07.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:07.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:08.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:08.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:08.838 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:08:08.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:08:08.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:08.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:09.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:10.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:10.178 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:08:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:08:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:10.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:11.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:11.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:11.431 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:08:11.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:08:11.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:11.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:11.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:12.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:12.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:12.863 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:08:12.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:08:12.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:12.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:13.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:14.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:14.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:14.104 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:08:14.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:08:14.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:14.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:14.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:15.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:15.703 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:08:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:08:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:16.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:17.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:17.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:17.146 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:08:17.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:08:17.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:17.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:17.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:18.688 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:08:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:08:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:18.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:18.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:19.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:19.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:19.930 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:08:19.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:08:19.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:19.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:20.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:08:21.495 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:08:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:21.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:22.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:22.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:22.867 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:08:22.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:08:22.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:22.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:23.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:24.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:24.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:24.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:08:24.677 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:08:24.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:24.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:25.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:26.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:26.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:26.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:08:26.153 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:08:26.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:26.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:08:26.397 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4184196370,v1:127.0.0.1:6811/4184196370] [v2:127.0.0.1:6812/4184196370,v1:127.0.0.1:6813/4184196370] exists,up 660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:08:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:08:26.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:08:26.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:08:26.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:08:26.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00bfb3c8-5964-483e-b7f8-08ed5e0d4873 2024-06-15T10:08:26.404 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 00bfb3c8-5964-483e-b7f8-08ed5e0d4873 2024-06-15T10:08:26.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 00bfb3c8-5964-483e-b7f8-08ed5e0d4873' 2024-06-15T10:08:26.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:08:26.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaaG1mWdzZGRAAwF2jaJk4uOiAZL9mfaZC5g== 2024-06-15T10:08:26.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaaG1mWdzZGRAAwF2jaJk4uOiAZL9mfaZC5g=="}' 2024-06-15T10:08:26.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00bfb3c8-5964-483e-b7f8-08ed5e0d4873 -i td/osd-scrub-repair/2/new.json 2024-06-15T10:08:26.922 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:08:26.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:08:26.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAaaG1mWdzZGRAAwF2jaJk4uOiAZL9mfaZC5g== --osd-uuid 00bfb3c8-5964-483e-b7f8-08ed5e0d4873 2024-06-15T10:08:26.984 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:26.987+0000 7f5edd0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:27.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.011+0000 7f5edd0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:27.008 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.011+0000 7f5edd0777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:27.008 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.011+0000 7f5edd0777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:08:27.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.011+0000 7f5edd0777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:08:27.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.011+0000 7f5edd0777c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:08:27.123 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:27.127+0000 7f5edd0777c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:08:36.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:08:36.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:08:36.728 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:08:36.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:08:36.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:08:37.002 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:08:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:08:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:08:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:08:37.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:08:37.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:08:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:37.043+0000 7fc949be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:37.087 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:37.091+0000 7fc949be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:37.089 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:37.091+0000 7fc949be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:08:37.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:37.235 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:08:37.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:08:37.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:37.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:37.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:37.648 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:37.651+0000 7fc949be17c0 -1 Falling back to public interface 2024-06-15T10:08:38.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:38.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:38.471 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:08:38.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:08:38.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:38.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:38.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:39.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:39.712 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:08:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:08:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:39.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:39.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:08:40.807+0000 7fc949be17c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:08:40.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:40.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:40.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:08:40.949 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:08:40.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:40.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:41.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:42.284 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:08:42.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:08:42.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:42.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:42.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:43.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:43.786 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:08:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:08:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:43.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:44.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:45.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:45.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:45.190 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:08:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:08:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:45.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:45.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:46.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:46.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:46.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:08:46.503 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:08:46.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:46.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:47.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:47.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:47.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:08:47.755 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:08:47.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:47.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:48.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:49.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:49.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:49.223 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:08:49.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:08:49.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:49.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:49.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:50.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:50.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:50.512 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:08:50.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:08:50.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:50.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:51.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:51.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:51.943 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:08:51.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:08:51.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:51.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:53.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:53.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:53.409 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:08:53.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:08:53.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:53.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:53.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:54.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:54.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:54.804 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:08:54.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:08:54.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:54.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:55.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:56.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:56.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:56.138 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:08:56.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:08:56.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:56.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:56.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:57.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:57.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:57.768 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:08:57.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:08:57.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:57.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:58.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:08:59.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:08:59.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:08:59.098 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:08:59.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:08:59.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:08:59.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:08:59.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:00.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:00.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:00.357 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:09:00.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:09:00.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:00.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:00.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:01.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:01.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:01.597 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:09:01.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:09:01.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:01.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:02.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:03.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:03.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:03.110 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:09:03.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:09:03.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:03.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:03.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:04.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:04.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:04.711 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:09:04.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:09:04.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:04.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:05.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:06.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:06.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:06.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:09:06.240 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:09:06.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:06.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:06.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:07.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:07.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:07.862 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:09:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:09:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:07.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:09.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:09.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:09.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:09:09.402 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:09:09.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:09.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:09.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:10.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:10.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:10.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:09:10.755 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:09:10.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:10.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:11.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:12.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:12.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:12.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:09:12.021 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:09:12.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:12.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:12.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:13.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:13.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:13.759 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:09:13.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:09:13.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:13.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:14.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:15.011 INFO:tasks.workunit.client.0.smithi116.stdout:27 2024-06-15T10:09:15.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 27 2024-06-15T10:09:15.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:15.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:09:15.251 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4111541740,v1:127.0.0.1:6819/4111541740] [v2:127.0.0.1:6820/4111541740,v1:127.0.0.1:6821/4111541740] exists,up 00bfb3c8-5964-483e-b7f8-08ed5e0d4873 2024-06-15T10:09:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:09:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:09:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:09:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-06-15T10:09:15.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:09:15.545 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:09:15.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:09:15.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:09:16.031 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:09:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:09:17.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:09:20.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-06-15T10:09:20.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:09:20.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:09:20.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:09:20.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:09:20.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:09:20.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:09:20.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:09:20.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:09:20.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:09:21.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:09:21.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:09:21.335 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:09:21.335 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:09:21.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:09:21.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:21.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:09:21.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-06-15T10:09:21.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-06-15T10:09:21.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-06-15T10:09:21.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:21.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:09:21.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411344 2024-06-15T10:09:21.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411344 2024-06-15T10:09:21.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-77309411344' 2024-06-15T10:09:21.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:21.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:09:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659144 2024-06-15T10:09:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659144 2024-06-15T10:09:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-77309411344 2-176093659144' 2024-06-15T10:09:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:21.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-06-15T10:09:21.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:21.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:09:21.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-06-15T10:09:21.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:21.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-06-15T10:09:21.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-06-15T10:09:21.771 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836504 2024-06-15T10:09:21.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:22.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-06-15T10:09:22.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:23.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:09:23.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:23.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-06-15T10:09:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:24.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:09:24.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:24.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-06-15T10:09:24.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:24.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411344 2024-06-15T10:09:24.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:24.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:09:24.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411344 2024-06-15T10:09:24.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411344 2024-06-15T10:09:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411344' 2024-06-15T10:09:24.494 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 77309411344 2024-06-15T10:09:24.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:09:24.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411344 -lt 77309411344 2024-06-15T10:09:24.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:24.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-176093659144 2024-06-15T10:09:24.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:24.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:09:24.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-176093659144 2024-06-15T10:09:24.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:24.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659144 2024-06-15T10:09:24.839 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 176093659144 2024-06-15T10:09:24.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 176093659144' 2024-06-15T10:09:24.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:09:25.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659144 -lt 176093659144 2024-06-15T10:09:25.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:09:25.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:25.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:09:25.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:09:25.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:09:25.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:09:25.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:09:25.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:09:25.568 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:25.568 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:25.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:09:25.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:09:25.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:09:25.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-15T10:09:26.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:09:26.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:09:27.188 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:09:27.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:09:28.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-15T10:09:28.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-15T10:09:28.712 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 allow_ec_overwrites to true 2024-06-15T10:09:28.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:09:28.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:09:28.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:09:28.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:09:28.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:09:28.732 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:09:28.732 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:09:28.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:09:28.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:09:28.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:09:28.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:09:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:09:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:09:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:09:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:09:29.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:29.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:09:29.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-15T10:09:29.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-15T10:09:29.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-15T10:09:29.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:29.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:09:29.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411346 2024-06-15T10:09:29.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411346 2024-06-15T10:09:29.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-77309411346' 2024-06-15T10:09:29.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:29.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:09:29.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659147 2024-06-15T10:09:29.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659147 2024-06-15T10:09:29.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-77309411346 2-176093659147' 2024-06-15T10:09:29.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:29.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-15T10:09:29.859 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836507 2024-06-15T10:09:29.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-15T10:09:29.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-15T10:09:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:31.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:09:31.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:31.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-15T10:09:31.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:09:32.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-06-15T10:09:32.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:32.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411346 2024-06-15T10:09:32.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:32.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:09:32.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411346 2024-06-15T10:09:32.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:32.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411346 2024-06-15T10:09:32.661 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 77309411346 2024-06-15T10:09:32.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411346' 2024-06-15T10:09:32.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:09:32.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411347 -lt 77309411346 2024-06-15T10:09:32.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:32.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-176093659147 2024-06-15T10:09:32.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:32.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:09:32.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-176093659147 2024-06-15T10:09:32.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:32.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659147 2024-06-15T10:09:32.897 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 176093659147 2024-06-15T10:09:32.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 176093659147' 2024-06-15T10:09:32.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:09:33.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659147 -lt 176093659147 2024-06-15T10:09:33.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:09:33.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:33.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:33.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:09:33.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:09:33.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:09:33.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:09:33.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:09:33.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:09:33.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:09:33.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:09:33.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:09:33.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:09:33.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:33.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-06-15T10:09:34.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:09:34.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-15T10:09:34.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-15T10:09:34.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:09:34.510 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-15T10:09:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:09:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:09:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:09:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:09:34.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-15T10:09:34.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:09:34.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-15T10:09:35.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:09:35.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:09:35.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:09:35.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:09:35.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:09:36.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:09:38.796 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:09:38.796 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-15T10:09:38.797 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:09:39.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:09:39.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:09:39.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:09:39.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:09:39.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:09:39.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:09:39.334 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:09:39.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:09:39.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:09:39.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:09:39.336 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:09:39.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:09:39.337 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:09:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:09:39.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:09:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:09:39.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:09:39.374 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:09:39.382+0000 7fca216147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:09:39.376 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:09:39.382+0000 7fca216147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:09:39.378 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:09:39.386+0000 7fca216147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:09:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:09:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:09:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:09:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:09:39.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:09:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:09:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:09:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:39.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:09:39.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:39.917 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:09:39.922+0000 7fca216147c0 -1 Falling back to public interface 2024-06-15T10:09:40.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:09:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:09:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:40.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:09:41.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:42.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:42.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:42.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:09:42.059 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:09:42.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:42.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:09:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:43.067 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:09:43.074+0000 7fca216147c0 -1 osd.1 65 log_to_monitors true 2024-06-15T10:09:43.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:43.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:43.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:09:43.303 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:09:43.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:43.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:09:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:09:44.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 73 up_thru 73 down_at 66 last_clean_interval [18,65) [v2:127.0.0.1:6810/3978828520,v1:127.0.0.1:6811/3978828520] [v2:127.0.0.1:6812/3978828520,v1:127.0.0.1:6813/3978828520] exists,up 660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:09:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:09:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:09:45.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:09:45.203 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:09:45.203 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:09:45.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:09:45.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:09:45.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:09:45.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:45.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:09:46.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-06-15T10:09:46.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-06-15T10:09:46.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-06-15T10:09:46.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:46.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:09:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-06-15T10:09:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-06-15T10:09:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-313532612610' 2024-06-15T10:09:46.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:46.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:09:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659151 2024-06-15T10:09:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659151 2024-06-15T10:09:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-313532612610 2-176093659151' 2024-06-15T10:09:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:46.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-06-15T10:09:46.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:09:46.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:46.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-06-15T10:09:46.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-06-15T10:09:46.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-06-15T10:09:46.815 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836511 2024-06-15T10:09:46.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:47.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-06-15T10:09:47.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:48.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:09:48.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:48.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836511 2024-06-15T10:09:48.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:48.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612610 2024-06-15T10:09:48.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:48.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:09:48.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612610 2024-06-15T10:09:48.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:48.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-06-15T10:09:48.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2024-06-15T10:09:48.333 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 313532612610 2024-06-15T10:09:48.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:09:48.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-06-15T10:09:48.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:48.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-176093659151 2024-06-15T10:09:48.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:48.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:09:48.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-176093659151 2024-06-15T10:09:48.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:48.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659151 2024-06-15T10:09:48.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 176093659151' 2024-06-15T10:09:48.585 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 176093659151 2024-06-15T10:09:48.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:09:48.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659151 -lt 176093659151 2024-06-15T10:09:48.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:09:48.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:48.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:49.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:09:49.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:09:49.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:09:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:09:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:09:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:09:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:09:49.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:09:49.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:09:49.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:09:49.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:49.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:49.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:09:49.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:09:49.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-06-15T10:09:49.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-06-15T10:09:49.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:49.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:49.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:49.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:50.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:50.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:50.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:09:34.897865+0000 '>' 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:50.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:09:51.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:09:51.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:09:51.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:09:51.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:51.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:51.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:51.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:51.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:09:34.897865+0000 '>' 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:51.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:09:52.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:09:52.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:09:52.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:09:52.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:52.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:52.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:52.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:53.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:09:34.897865+0000 '>' 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:53.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:09:54.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:09:54.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:09:54.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:09:54.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:54.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:54.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:54.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:54.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:09:34.897865+0000 '>' 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:54.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:09:55.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:09:55.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:09:55.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:09:55.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:09:55.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:09:55.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:09:55.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:09:55.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:09:49.972848+0000 '>' 2024-06-15T10:09:34.897865+0000 2024-06-15T10:09:55.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:09:55.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-06-15T10:09:55.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:09:55.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:09:55.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:09:55.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:09:55.522 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:09:55.522 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:09:55.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:09:55.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:09:55.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:09:55.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:09:55.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:09:55.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:09:55.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:09:55.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:09:55.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:09:55.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:09:55.873 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:09:55.873 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:09:55.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:09:55.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:55.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:09:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-06-15T10:09:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-06-15T10:09:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-06-15T10:09:56.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:56.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:09:56.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612613 2024-06-15T10:09:56.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612613 2024-06-15T10:09:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-313532612613' 2024-06-15T10:09:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:09:56.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:09:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659154 2024-06-15T10:09:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659154 2024-06-15T10:09:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-313532612613 2-176093659154' 2024-06-15T10:09:56.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:56.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:56.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-06-15T10:09:56.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:09:56.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-06-15T10:09:56.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-06-15T10:09:56.450 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836514 2024-06-15T10:09:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-06-15T10:09:56.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:56.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-06-15T10:09:56.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:09:57.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:09:57.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:09:58.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836514 2024-06-15T10:09:58.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:58.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612613 2024-06-15T10:09:58.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:58.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:09:58.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612613 2024-06-15T10:09:58.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:58.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612613 2024-06-15T10:09:58.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612613' 2024-06-15T10:09:58.026 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 313532612613 2024-06-15T10:09:58.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:09:58.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612613 -lt 313532612613 2024-06-15T10:09:58.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:09:58.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:09:58.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-176093659154 2024-06-15T10:09:58.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:09:58.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-176093659154 2024-06-15T10:09:58.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:09:58.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659154 2024-06-15T10:09:58.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 176093659154' 2024-06-15T10:09:58.294 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 176093659154 2024-06-15T10:09:58.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:09:58.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659154 -lt 176093659154 2024-06-15T10:09:58.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:09:58.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:58.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:58.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:09:58.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:09:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:09:59.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:09:59.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:09:59.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:09:59.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:09:59.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:09:59.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:09:59.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-06-15T10:09:59.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-06-15T10:09:59.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:09:59.417 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-06-15T10:09:59.418 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:09:59.418 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:09:59.418 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:09:59.418 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:09:59.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-06-15T10:09:59.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:09:59.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-06-15T10:10:00.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:10:00.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:10:00.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:10:00.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:10:00.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:10:00.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:10:00.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:10:00.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:10:00.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:10:00.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:10:00.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:10:00.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:10:03.108 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:10:03.108 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-06-15T10:10:03.108 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:10:03.108 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:10:03.109 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:10:03.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:10:03.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:10:03.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:10:03.400 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:10:03.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:10:03.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-06-15T10:10:03.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:10:03.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:10:03.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:10:03.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:10:03.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:10:03.439 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:03.446+0000 7fc8008e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:03.458 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:03.466+0000 7fc8008e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:03.460 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:03.466+0000 7fc8008e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:10:03.877 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:10:03.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:03.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:10:04.011 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:04.018+0000 7fc8008e47c0 -1 Falling back to public interface 2024-06-15T10:10:04.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:05.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:05.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:05.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:10:05.181 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:10:05.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:05.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:10:05.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:06.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:06.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:06.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:10:06.464 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:10:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:10:06.646 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:06.654+0000 7fc8008e47c0 -1 osd.1 85 log_to_monitors true 2024-06-15T10:10:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:07.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:10:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:10:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:10:08.055 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 86 last_clean_interval [73,85) [v2:127.0.0.1:6810/750438960,v1:127.0.0.1:6811/750438960] [v2:127.0.0.1:6812/750438960,v1:127.0.0.1:6813/750438960] exists,up 660ce8c9-9eea-48f4-948b-b062306b7a64 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:10:08.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:10:08.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:10:08.057 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:10:08.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:10:08.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:10:08.058 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:10:08.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:10:08.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:10:08.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:10:08.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:10:08.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:10:08.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:10:08.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:10:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:10:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:10:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:10:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:10:08.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:10:09.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-06-15T10:10:09.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-06-15T10:10:09.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-06-15T10:10:09.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:10:09.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:10:09.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-06-15T10:10:09.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-06-15T10:10:09.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-395136991234' 2024-06-15T10:10:09.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:10:09.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:10:09.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659157 2024-06-15T10:10:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659157 2024-06-15T10:10:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-395136991234 2-176093659157' 2024-06-15T10:10:09.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:10:09.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-06-15T10:10:09.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:10:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:10:09.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-06-15T10:10:09.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:10:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-06-15T10:10:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-06-15T10:10:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836518 2024-06-15T10:10:09.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:10:10.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-06-15T10:10:10.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:10:10.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-06-15T10:10:10.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:10:10.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:10:10.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-06-15T10:10:10.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:10:10.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-06-15T10:10:10.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-06-15T10:10:10.093 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 395136991234 2024-06-15T10:10:10.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:10:10.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 395136991234 2024-06-15T10:10:10.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:10:11.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:10:11.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:10:11.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-06-15T10:10:11.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:10:11.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:10:11.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-176093659157 2024-06-15T10:10:11.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:10:11.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-176093659157 2024-06-15T10:10:11.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:10:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659157 2024-06-15T10:10:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 176093659157' 2024-06-15T10:10:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 176093659157 2024-06-15T10:10:11.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:10:11.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659158 -lt 176093659157 2024-06-15T10:10:11.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:10:11.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:10:11.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:10:12.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:10:12.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:10:12.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:10:12.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:10:12.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:10:12.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:10:12.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:10:12.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:10:12.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:10:12.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:10:12.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:10:12.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:10:12.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:10:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:10:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:10:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:10:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:10:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:10:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:10:12.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:10:12.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:10:12.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:10:12.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:10:12.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:10:12.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:10:12.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:10:12.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:10:12.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:10:12.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:10:12.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:10:13.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:10:13.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:10:13.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:10:13.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:10:13.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:10:13.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:10:13.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:10:13.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:10:13.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:10:13.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:10:13.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:10:13.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:10:13.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:10:13.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:10:13.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:10:13.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:10:13.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:10:13.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:10:13.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:10:13.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:10:13.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:10:13.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:10:13.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:10:13.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:10:13.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:10:13.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:10:13.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:10:13.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:10:13.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:10:13.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:13.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:13.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:10:13.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:10:13.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:10:13.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:10:13.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:10:13.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:10:13.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:10:13.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:10:13.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:10:13.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:10:13.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:10:13.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:10:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:10:13.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:10:13.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:10:13.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:10:13.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:10:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:10:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:10:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:10:13.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:10:13.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:10:13.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:10:13.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:10:13.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:10:13.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:13.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:10:13.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:10:13.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:10:13.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:10:13.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:10:13.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:13.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:13.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:10:13.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-15T10:10:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:10:13.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:10:14.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:10:14.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:10:14.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:10:14.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:10:14.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:14.139 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:14.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:14.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:10:14.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:10:14.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:10:14.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:10:14.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:10:14.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:10:14.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:10:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:10:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:10:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:10:14.402 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:10:14.403 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:14.403 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:14.403 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:10:14.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:10:14.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:10:14.524 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:14.525 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:10:14.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:10:14.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:10:14.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-15T10:10:14.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:10:14.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:10:14.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:10:14.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:10:14.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:10:14.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:15.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:15.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:10:15.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:10:15.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-15T10:10:15.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:10:15.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:10:15.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:15.572 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:10:15.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:10:15.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:10:15.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:10:15.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77cce5dc-f143-4877-8859-941ba36d72b0 2024-06-15T10:10:15.575 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 77cce5dc-f143-4877-8859-941ba36d72b0 2024-06-15T10:10:15.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 77cce5dc-f143-4877-8859-941ba36d72b0' 2024-06-15T10:10:15.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:10:15.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHaG1m/AC0IxAA1u7O4DBHc8TGtqeG6k+w4Q== 2024-06-15T10:10:15.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHaG1m/AC0IxAA1u7O4DBHc8TGtqeG6k+w4Q=="}' 2024-06-15T10:10:15.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77cce5dc-f143-4877-8859-941ba36d72b0 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:10:15.919 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:10:15.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:10:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHaG1m/AC0IxAA1u7O4DBHc8TGtqeG6k+w4Q== --osd-uuid 77cce5dc-f143-4877-8859-941ba36d72b0 2024-06-15T10:10:15.973 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:15.978+0000 7fa8b7c9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:16.003 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.010+0000 7fa8b7c9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:16.005 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.010+0000 7fa8b7c9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:16.005 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.010+0000 7fa8b7c9c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:10:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.010+0000 7fa8b7c9c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:10:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.010+0000 7fa8b7c9c7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:10:16.103 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:16.110+0000 7fa8b7c9c7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:10:23.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:10:23.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:10:23.961 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:10:23.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:10:23.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:10:24.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:10:24.234 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:10:24.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:10:24.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:10:24.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:10:24.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:10:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:24.269+0000 7f79c16797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:24.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:24.321+0000 7f79c16797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:24.318 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:24.325+0000 7f79c16797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:10:24.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:24.456 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:10:24.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:10:24.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:24.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:24.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:24.889 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:24.893+0000 7f79c16797c0 -1 Falling back to public interface 2024-06-15T10:10:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:25.704 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:10:25.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:10:25.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:25.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:25.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:26.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:26.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:26.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:10:26.946 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:10:26.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:26.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:27.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:27.760 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:27.765+0000 7f79c16797c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:10:28.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:28.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:28.186 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:10:28.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:10:28.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:28.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:28.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:29.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:29.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:29.583 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:10:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:10:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:30.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:30.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:30.913 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:10:30.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:10:30.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:30.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:31.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:32.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:32.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:32.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:10:32.152 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:10:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:32.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:33.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:33.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:33.488 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:10:33.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:10:33.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:33.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:34.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:34.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:34.841 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:10:34.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:10:34.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:34.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:35.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:36.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:36.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:36.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:10:36.083 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:10:36.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:36.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:36.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:37.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:37.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:37.324 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:10:37.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:10:37.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:37.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:37.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:38.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:38.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:38.739 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:10:38.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:10:38.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:38.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:38.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:39.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:39.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:39.982 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:10:39.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:10:39.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:39.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:40.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:41.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:41.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:41.397 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:10:41.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:10:41.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:41.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:41.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:42.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:42.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:42.755 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:10:42.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:10:42.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:42.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:43.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:44.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:44.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:44.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:10:44.121 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:10:44.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:44.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:44.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:45.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:45.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:45.410 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:10:45.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:10:45.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:45.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:45.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:46.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:46.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:46.656 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:10:46.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:10:46.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:46.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:47.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:48.517 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:10:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:10:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:48.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:49.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:49.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:49.981 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:10:49.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:10:49.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:49.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:50.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:51.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:51.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:51.283 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:10:51.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:10:51.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:51.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:51.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:52.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:52.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:52.528 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:10:52.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:10:52.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:52.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:53.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:54.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:54.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:54.155 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:10:54.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:10:54.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:54.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:55.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:55.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:55.827 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:10:55.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:10:55.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:55.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:56.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:57.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:57.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:57.414 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:10:57.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:10:57.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:57.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:58.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:10:59.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:10:59.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:10:59.043 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:10:59.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:10:59.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:10:59.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:10:59.378 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2749650688,v1:127.0.0.1:6803/2749650688] [v2:127.0.0.1:6804/2749650688,v1:127.0.0.1:6805/2749650688] exists,up 77cce5dc-f143-4877-8859-941ba36d72b0 2024-06-15T10:10:59.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:10:59.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:10:59.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:10:59.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:10:59.383 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:10:59.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:10:59.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:10:59.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:10:59.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:10:59.385 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:10:59.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 143ad697-a0d4-467c-8fb2-1f0f68eb917e' 2024-06-15T10:10:59.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:10:59.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzaG1mytWXGBAAVz+BI+kyg3OWTsl1CmJ1DA== 2024-06-15T10:10:59.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzaG1mytWXGBAAVz+BI+kyg3OWTsl1CmJ1DA=="}' 2024-06-15T10:10:59.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 143ad697-a0d4-467c-8fb2-1f0f68eb917e -i td/osd-scrub-repair/1/new.json 2024-06-15T10:10:59.738 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:10:59.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:10:59.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzaG1mytWXGBAAVz+BI+kyg3OWTsl1CmJ1DA== --osd-uuid 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:10:59.786 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.793+0000 7f4e32bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:59.874 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.881+0000 7f4e32bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:59.876 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.881+0000 7f4e32bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:10:59.877 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.881+0000 7f4e32bc47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:10:59.877 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.881+0000 7f4e32bc47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:10:59.877 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.881+0000 7f4e32bc47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:10:59.959 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:10:59.965+0000 7f4e32bc47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:11:08.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:11:08.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:11:08.122 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:11:08.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:11:08.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:11:08.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:11:08.395 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:11:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:11:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:11:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:11:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:11:08.427 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:08.432+0000 7f8f1f97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:08.478 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:08.484+0000 7f8f1f97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:08.482 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:08.484+0000 7f8f1f97c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:11:08.625 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:11:08.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:08.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:08.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:09.045 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:09.048+0000 7f8f1f97c7c0 -1 Falling back to public interface 2024-06-15T10:11:09.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:09.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:09.878 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:11:09.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:11:09.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:09.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:10.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:11.118 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:11:11.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:11:11.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:11.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:11.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:12.017 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:12.020+0000 7f8f1f97c7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:11:12.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:12.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:12.361 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:11:12.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:11:12.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:12.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:12.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:13.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:13.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:13.950 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:11:13.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:11:13.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:13.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:14.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:15.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:15.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:15.329 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:11:15.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:11:15.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:15.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:15.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:16.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:16.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:16.572 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:11:16.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:11:16.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:16.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:16.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:17.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:17.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:17.819 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:11:17.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:11:17.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:17.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:18.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:19.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:19.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:19.291 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:11:19.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:11:19.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:19.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:19.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:20.535 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:11:20.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:11:20.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:20.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:21.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:22.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:22.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:22.007 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:11:22.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:11:22.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:22.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:22.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:23.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:23.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:23.355 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:11:23.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:11:23.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:23.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:23.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:24.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:24.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:24.600 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:11:24.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:11:24.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:24.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:25.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:26.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:26.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:26.021 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:11:26.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:11:26.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:26.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:26.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:27.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:27.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:27.326 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:11:27.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:11:27.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:27.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:28.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:28.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:28.562 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:11:28.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:11:28.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:28.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:28.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:29.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:30.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:30.000 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:11:30.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:11:30.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:30.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:30.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:31.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:31.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:31.483 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:11:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:11:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:31.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:32.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:32.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:32.856 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:11:32.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:11:32.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:32.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:33.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:34.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:34.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:34.560 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:11:34.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:11:34.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:34.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:35.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:36.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:36.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:36.289 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:11:36.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:11:36.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:36.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:36.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:37.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:37.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:37.910 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:11:37.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:11:37.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:37.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:38.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:39.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:39.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:39.569 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:11:39.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:11:39.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:39.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:40.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:41.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:41.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:41.269 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:11:41.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:11:41.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:41.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:42.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:42.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:42.912 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:11:42.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:11:42.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:42.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:11:43.173 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3589324933,v1:127.0.0.1:6811/3589324933] [v2:127.0.0.1:6812/3589324933,v1:127.0.0.1:6813/3589324933] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:11:43.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:11:43.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:11:43.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:11:43.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:11:43.181 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:11:43.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 25912c94-7f79-4981-a94a-a32e3e3aeeb2' 2024-06-15T10:11:43.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:11:43.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfaG1mVN9DDBAAm4CMGhbNjVL1jVjWMb7PDQ== 2024-06-15T10:11:43.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfaG1mVN9DDBAAm4CMGhbNjVL1jVjWMb7PDQ=="}' 2024-06-15T10:11:43.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25912c94-7f79-4981-a94a-a32e3e3aeeb2 -i td/osd-scrub-repair/2/new.json 2024-06-15T10:11:43.609 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:11:43.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:11:43.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfaG1mVN9DDBAAm4CMGhbNjVL1jVjWMb7PDQ== --osd-uuid 25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:11:43.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.663+0000 7febf8b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:43.744 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.747+0000 7febf8b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:43.745 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.747+0000 7febf8b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:43.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.747+0000 7febf8b677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:11:43.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.747+0000 7febf8b677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:11:43.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.747+0000 7febf8b677c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:11:43.835 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:43.839+0000 7febf8b677c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:11:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:11:51.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:11:51.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:11:51.765 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:11:51.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:11:52.034 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:11:52.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:11:52.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:11:52.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:11:52.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:11:52.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:11:52.066 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:52.067+0000 7f44c69fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:52.068 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:52.071+0000 7f44c69fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:52.069 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:52.071+0000 7f44c69fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:11:52.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:11:52.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:11:52.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:11:52.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:11:52.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:11:52.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:52.257 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:11:52.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:11:52.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:52.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:52.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:52.641 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:52.643+0000 7f44c69fd7c0 -1 Falling back to public interface 2024-06-15T10:11:53.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:53.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:53.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:11:53.514 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:11:53.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:53.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:53.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:54.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:54.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:54.747 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:11:54.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:11:54.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:54.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:54.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:55.598 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:11:55.599+0000 7f44c69fd7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:11:55.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:55.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:55.990 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:11:55.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:11:55.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:55.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:56.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:57.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:57.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:57.617 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:11:57.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:11:57.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:57.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:58.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:11:59.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:11:59.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:11:59.091 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:11:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:11:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:11:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:11:59.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:00.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:12:00.527 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:12:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:00.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:01.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:01.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:01.933 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:12:01.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:12:01.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:01.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:02.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:03.179 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:12:03.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:12:03.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:03.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:03.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:04.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:04.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:04.582 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:12:04.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:12:04.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:04.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:04.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:05.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:05.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:05.843 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:12:05.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:12:05.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:05.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:06.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:07.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:07.156 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:12:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:12:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:07.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:12:08.401 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:12:08.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:08.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:08.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:12:09.808 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:12:09.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:09.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:10.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:11.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:11.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:11.048 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:12:11.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:12:11.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:11.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:11.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:12.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:12.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:12.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:12:12.473 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:12:12.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:12.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:12.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:13.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:13.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:13.786 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:12:13.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:12:13.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:13.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:14.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:15.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:15.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:15.029 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:12:15.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:12:15.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:15.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:15.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:16.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:16.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:16.273 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:12:16.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:12:16.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:16.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:16.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:17.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:17.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:17.820 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:12:17.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:12:17.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:17.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:18.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:19.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:19.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:19.060 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:12:19.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:12:19.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:19.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:19.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:20.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:20.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:20.304 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:12:20.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:12:20.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:20.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:20.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:21.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:21.908 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:12:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:12:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:21.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:22.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:23.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:23.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:23.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:12:23.198 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:12:23.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:23.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:23.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:24.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:24.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:24.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:12:24.852 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:12:24.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:24.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:26.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:26.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:26.093 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:12:26.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:12:26.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:26.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:26.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:27.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:27.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:27.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:12:27.472 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:12:27.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:27.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:12:27.712 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1040911912,v1:127.0.0.1:6819/1040911912] [v2:127.0.0.1:6820/1040911912,v1:127.0.0.1:6821/1040911912] exists,up 25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:12:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:12:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:12:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:12:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:12:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:12:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:12:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:12:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bb180a4a-91ab-436e-b208-82a0e618b076' 2024-06-15T10:12:27.720 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:12:27.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:12:27.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALaW1mpMpsLBAA+oTDhueK0ziwAc1z6mkXJw== 2024-06-15T10:12:27.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALaW1mpMpsLBAA+oTDhueK0ziwAc1z6mkXJw=="}' 2024-06-15T10:12:27.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb180a4a-91ab-436e-b208-82a0e618b076 -i td/osd-scrub-repair/3/new.json 2024-06-15T10:12:28.114 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:12:28.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-15T10:12:28.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALaW1mpMpsLBAA+oTDhueK0ziwAc1z6mkXJw== --osd-uuid bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:12:28.174 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.178+0000 7fd5e613f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:28.277 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.282+0000 7fd5e613f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:28.279 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.286+0000 7fd5e613f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:28.280 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.286+0000 7fd5e613f7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:12:28.280 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.286+0000 7fd5e613f7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:12:28.280 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.286+0000 7fd5e613f7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:12:28.331 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:28.338+0000 7fd5e613f7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-15T10:12:36.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-15T10:12:36.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:12:36.486 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T10:12:36.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T10:12:36.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:12:36.790 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T10:12:36.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T10:12:36.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:12:36.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:12:36.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:12:36.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:12:36.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:36.826+0000 7ff8a5a8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:36.825 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:36.830+0000 7ff8a5a8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:36.826 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:36.830+0000 7ff8a5a8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:12:37.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T10:12:37.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:12:37.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:12:37.027 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:12:37.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:12:37.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:12:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:12:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:37.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:37.381 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:37.386+0000 7ff8a5a8b7c0 -1 Falling back to public interface 2024-06-15T10:12:38.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:38.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:38.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:12:38.261 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:12:38.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:38.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:38.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:39.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:39.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:39.503 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:12:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:12:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:39.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:12:40.378+0000 7ff8a5a8b7c0 -1 osd.3 0 log_to_monitors true 2024-06-15T10:12:40.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:40.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:40.744 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:12:40.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:12:40.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:40.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:41.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:42.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:42.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:42.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:12:42.366 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:12:42.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:42.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:42.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:43.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:43.620 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:12:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:12:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:44.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:44.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:44.914 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:12:44.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:12:44.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:44.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:45.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:46.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:46.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:46.251 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:12:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:12:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:46.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:46.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:47.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:12:47.638 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:12:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:47.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:47.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:48.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:48.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:48.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:12:48.960 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:12:48.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:48.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:49.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:50.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:50.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:50.339 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:12:50.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:12:50.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:50.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:50.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:51.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:51.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:51.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:12:51.762 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:12:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:51.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:52.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:53.061 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:12:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:12:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:53.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:54.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:54.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:54.522 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:12:54.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:12:54.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:54.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:54.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:55.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:55.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:55.765 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:12:55.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:12:55.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:55.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:56.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:57.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:57.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:57.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:12:57.036 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:12:57.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:57.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:57.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:58.520 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:12:58.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:12:58.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:58.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:12:58.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:12:59.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:12:59.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:12:59.864 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:12:59.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:12:59.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:12:59.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:00.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:01.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:01.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:01.748 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:13:01.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:13:01.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:01.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:02.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:03.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:03.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:03.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:13:03.405 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:13:03.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:03.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:04.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:05.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:05.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:05.061 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:13:05.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:13:05.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:05.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:05.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:06.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:06.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:06.665 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:13:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:13:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:06.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:07.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:08.124 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:13:08.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:13:08.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:08.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:08.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:09.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:09.587 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:13:09.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:13:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:09.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:09.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:10.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:10.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:10.843 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:13:10.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:13:10.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:10.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:11.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:12.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:12.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:12.191 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:13:12.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:13:12.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:12.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:12.430 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2813415690,v1:127.0.0.1:6827/2813415690] [v2:127.0.0.1:6828/2813415690,v1:127.0.0.1:6829/2813415690] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:13:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:13:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:13:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:13:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-15T10:13:12.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:13:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:13:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:13:12.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:13:13.255 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:13:13.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:13:14.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:13:17.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-15T10:13:17.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:13:17.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:13:17.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:13:17.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:13:17.381 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:13:17.382 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:13:17.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:13:17.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:13:17.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:13:17.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:13:17.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:17.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:13:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-06-15T10:13:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-06-15T10:13:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-06-15T10:13:17.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:17.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:13:18.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-06-15T10:13:18.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-06-15T10:13:18.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672986' 2024-06-15T10:13:18.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:18.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:13:18.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-15T10:13:18.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-15T10:13:18.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672986 2-64424509453' 2024-06-15T10:13:18.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:18.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:13:18.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-06-15T10:13:18.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-06-15T10:13:18.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672986 2-64424509453 3-85899345925' 2024-06-15T10:13:18.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-06-15T10:13:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:18.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:13:18.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-06-15T10:13:18.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:18.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-06-15T10:13:18.440 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836513 2024-06-15T10:13:18.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-06-15T10:13:18.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:18.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-06-15T10:13:18.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:13:19.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:13:19.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:19.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836513 2024-06-15T10:13:19.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:19.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-06-15T10:13:20.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:20.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:13:20.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-06-15T10:13:20.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:20.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-06-15T10:13:20.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-06-15T10:13:20.004 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672986 2024-06-15T10:13:20.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:13:20.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-06-15T10:13:20.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:20.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-15T10:13:20.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:13:20.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-15T10:13:20.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:20.262 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509453 2024-06-15T10:13:20.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-15T10:13:20.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-15T10:13:20.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:13:20.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-06-15T10:13:20.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:20.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345925 2024-06-15T10:13:20.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:20.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:13:20.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345925 2024-06-15T10:13:20.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:20.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-06-15T10:13:20.540 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345925 2024-06-15T10:13:20.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-06-15T10:13:20.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:13:20.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-06-15T10:13:20.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:13:20.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:20.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:21.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:13:21.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:21.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:21.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:13:21.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:21.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:21.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:13:21.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-15T10:13:22.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:13:22.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:13:22.674 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:13:22.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:13:23.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T10:13:23.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:13:23.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:13:23.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:13:23.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:13:23.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:13:23.703 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:13:23.703 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:13:23.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:13:23.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:13:23.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:13:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:13:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:13:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:13:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:13:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:13:23.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:24.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:13:24.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-06-15T10:13:24.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-06-15T10:13:24.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-06-15T10:13:24.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:24.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:13:24.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-06-15T10:13:24.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-06-15T10:13:24.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989' 2024-06-15T10:13:24.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:24.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:13:24.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-06-15T10:13:24.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-06-15T10:13:24.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509455' 2024-06-15T10:13:24.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:24.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:13:25.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-06-15T10:13:25.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-06-15T10:13:25.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509455 3-85899345927' 2024-06-15T10:13:25.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:25.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-06-15T10:13:25.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:25.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:13:25.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-06-15T10:13:25.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-06-15T10:13:25.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-15T10:13:25.040 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836516 2024-06-15T10:13:25.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:25.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-06-15T10:13:25.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:13:26.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:13:26.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:26.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-06-15T10:13:26.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:26.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-06-15T10:13:26.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:26.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:13:26.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-06-15T10:13:26.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:26.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-06-15T10:13:26.539 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672989 2024-06-15T10:13:26.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-15T10:13:26.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:13:26.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-15T10:13:26.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:26.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-06-15T10:13:26.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:26.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:13:26.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-06-15T10:13:26.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:26.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-06-15T10:13:26.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-06-15T10:13:26.828 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509455 2024-06-15T10:13:26.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:13:27.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-06-15T10:13:27.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:27.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345927 2024-06-15T10:13:27.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:27.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:13:27.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345927 2024-06-15T10:13:27.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:27.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-06-15T10:13:27.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-06-15T10:13:27.103 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345927 2024-06-15T10:13:27.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:13:27.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345927 2024-06-15T10:13:27.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:13:27.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:27.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:13:27.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:27.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:27.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:27.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:27.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:27.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:27.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:27.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:13:27.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:27.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:27.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:13:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:13:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:13:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:13:28.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:13:28.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:13:28.727 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:13:28.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:13:29.094 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:13:29.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:13:29.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:13:29.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:13:29.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-15T10:13:29.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:13:29.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:13:29.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:13:29.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:13:29.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-15T10:13:29.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-15T10:13:29.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-15T10:13:29.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T10:13:29.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-15T10:13:29.924 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:13:29.924 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:13:30.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-15T10:13:30.235 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:30.235 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:30.235 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:13:30.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:13:30.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:13:30.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:13:30.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:13:30.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:13:30.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:13:30.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:13:33.129 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:13:33.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:13:33.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:13:33.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:13:33.668 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:13:33.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:13:33.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:13:33.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:13:33.670 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:13:33.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:13:33.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:13:33.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:13:33.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:13:33.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:13:33.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:13:33.703 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:33.705+0000 7f395a4f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:33.705 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:33.709+0000 7f395a4f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:33.707 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:33.709+0000 7f395a4f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:33.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:13:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:13:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:33.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:34.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:34.246 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:34.249+0000 7f395a4f67c0 -1 Falling back to public interface 2024-06-15T10:13:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:13:35.180 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:35.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:35.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:35.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:36.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:13:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:36.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:37.120 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:37.124+0000 7f395a4f67c0 -1 osd.3 36 log_to_monitors true 2024-06-15T10:13:37.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:37.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:37.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:13:37.671 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:13:37.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:37.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:38.023 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 40 up_thru 31 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/690874360,v1:127.0.0.1:6827/690874360] [v2:127.0.0.1:6828/690874360,v1:127.0.0.1:6829/690874360] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:13:38.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:13:38.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:13:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:13:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:13:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:13:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:13:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:13:38.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:13:38.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:38.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:13:38.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-06-15T10:13:38.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-06-15T10:13:38.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-06-15T10:13:38.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:38.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:13:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672993 2024-06-15T10:13:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672993 2024-06-15T10:13:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672993' 2024-06-15T10:13:38.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:38.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:13:39.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-15T10:13:39.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-15T10:13:39.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672993 2-64424509459' 2024-06-15T10:13:39.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:13:39.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:13:39.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T10:13:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T10:13:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672993 2-64424509459 3-171798691842' 2024-06-15T10:13:39.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:39.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-06-15T10:13:39.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:39.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:13:39.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-06-15T10:13:39.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-06-15T10:13:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-06-15T10:13:39.389 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836519 2024-06-15T10:13:39.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:39.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-06-15T10:13:39.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:13:40.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:13:40.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:13:40.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836519 2024-06-15T10:13:40.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:40.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672993 2024-06-15T10:13:40.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:13:40.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672993 2024-06-15T10:13:40.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:40.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672993 2024-06-15T10:13:40.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-06-15T10:13:40.959 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672993 2024-06-15T10:13:40.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:13:41.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672993 -lt 42949672993 2024-06-15T10:13:41.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:41.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:41.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-15T10:13:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:13:41.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-15T10:13:41.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-15T10:13:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-15T10:13:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509459 2024-06-15T10:13:41.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:13:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-06-15T10:13:41.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:13:41.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-06-15T10:13:41.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:13:41.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:13:41.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-06-15T10:13:41.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:13:41.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T10:13:41.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-06-15T10:13:41.511 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 171798691842 2024-06-15T10:13:41.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:13:41.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-15T10:13:41.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:13:41.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:41.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:42.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:13:42.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:42.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:42.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:42.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:42.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:42.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:42.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:42.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:13:42.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:42.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:42.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:13:42.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:13:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:13:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:42.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:42.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:42.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:42.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:42.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:42.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:42.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:13:42.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:42.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:42.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:43.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:13:43.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:13:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:13:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:13:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:13:43.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:13:43.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:13:43.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:13:43.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:43.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:43.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:43.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:43.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:43.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:43.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:43.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:13:43.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:43.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:43.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:44.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:13:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-06-15T10:13:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:13:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:13:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:13:44.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:13:44.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:13:44.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:44.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:44.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:44.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:44.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:44.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:44.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:44.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:13:44.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:44.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:44.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:44.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:13:44.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:13:44.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:13:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:13:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:13:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:13:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:13:45.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:13:45.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:13:45.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:45.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:45.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:45.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:45.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:45.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:45.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:45.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:13:45.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:45.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:45.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:45.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:13:45.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:13:45.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:13:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:13:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:13:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:13:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:13:46.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:13:46.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:13:46.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:13:46.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:13:46.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:13:46.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:13:46.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:13:46.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:13:46.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:13:46.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:13:46.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:13:46.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:13:46.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:13:46.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:13:46.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:13:46.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:13:46.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:13:46.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:13:46.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:13:46.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:13:46.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:13:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:13:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:13:47.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:13:47.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:13:47.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:47.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:47.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:47.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:47.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:47.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:13:47.620 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:13:47.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:47.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:47.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:47.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:47.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:47.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:47.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:22.497402+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:47.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:13:48.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:13:48.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:48.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:48.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:48.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:48.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:48.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:49.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:22.497402+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:49.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:13:50.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:13:50.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:50.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:50.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:50.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:50.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:50.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:50.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:22.497402+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:50.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:13:51.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:13:51.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:51.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:51.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:51.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:51.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:51.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:51.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:22.497402+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:51.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:13:52.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:13:52.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:52.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:52.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:52.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:52.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:52.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:52.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:22.497402+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:52.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:13:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:13:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:13:53.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:13:53.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:13:53.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:13:53.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:13:53.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:13:54.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:48.359637+0000 '>' 2024-06-15T10:13:22.497402+0000 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:13:54.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:13:54.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:13:54.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:13:54.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:13:54.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:13:54.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:13:54.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:13:54.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:13:54.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:13:54.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:13:54.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:13:54.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:13:54.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:13:54.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:13:56.588 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:13:56.588 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:13:56.588 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:13:56.589 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:13:57.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:13:57.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:13:57.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:13:57.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:13:57.129 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:13:57.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:13:57.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:13:57.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:13:57.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:13:57.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:13:57.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:13:57.167 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:57.168+0000 7f945a2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:57.170 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:57.172+0000 7f945a2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:57.176+0000 7f945a2bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:13:57.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:13:57.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:13:57.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:57.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:57.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:57.707 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:13:57.708+0000 7f945a2bc7c0 -1 Falling back to public interface 2024-06-15T10:13:58.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:58.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:58.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:13:58.598 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:13:58.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:58.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:13:58.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:13:59.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:13:59.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:13:59.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:13:59.838 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:13:59.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:13:59.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:00.617 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:00.620+0000 7f945a2bc7c0 -1 osd.3 42 log_to_monitors true 2024-06-15T10:14:01.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:01.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:01.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:14:01.256 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:14:01.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:01.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:01.868 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/2929586632,v1:127.0.0.1:6827/2929586632] [v2:127.0.0.1:6828/2929586632,v1:127.0.0.1:6829/2929586632] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:14:01.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:14:01.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:14:01.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:14:01.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:14:01.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:14:01.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:14:01.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:14:01.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:14:01.871 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:14:01.871 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:14:01.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:14:01.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:14:01.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:14:01.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:14:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:02.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:14:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-06-15T10:14:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-06-15T10:14:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-06-15T10:14:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:02.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:14:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672998 2024-06-15T10:14:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672998 2024-06-15T10:14:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672998' 2024-06-15T10:14:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:02.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:14:03.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-06-15T10:14:03.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-06-15T10:14:03.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672998 2-64424509465' 2024-06-15T10:14:03.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:03.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:14:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-06-15T10:14:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-06-15T10:14:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672998 2-64424509465 3-197568495618' 2024-06-15T10:14:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:03.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-06-15T10:14:03.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:03.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:14:03.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-06-15T10:14:03.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:03.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-06-15T10:14:03.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-06-15T10:14:03.193 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836525 2024-06-15T10:14:03.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:03.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-06-15T10:14:03.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:14:04.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:14:04.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:04.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-06-15T10:14:04.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:04.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672998 2024-06-15T10:14:04.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:04.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:14:04.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672998 2024-06-15T10:14:04.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672998 2024-06-15T10:14:04.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-06-15T10:14:04.666 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672998 2024-06-15T10:14:04.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:14:04.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672998 -lt 42949672998 2024-06-15T10:14:04.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:04.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-06-15T10:14:04.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:04.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:14:04.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-06-15T10:14:04.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:04.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-06-15T10:14:04.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-06-15T10:14:04.906 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509465 2024-06-15T10:14:04.906 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:14:05.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-06-15T10:14:05.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:05.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-06-15T10:14:05.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:05.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:14:05.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-06-15T10:14:05.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:05.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-06-15T10:14:05.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-06-15T10:14:05.130 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 197568495618 2024-06-15T10:14:05.130 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:14:05.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-06-15T10:14:05.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:14:05.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:05.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:05.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:14:05.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:05.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:05.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:14:05.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:05.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:05.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:06.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:14:06.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:14:06.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:14:06.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:14:06.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:06.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:14:06.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:06.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:06.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:06.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:06.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:14:06.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:06.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:06.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:06.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:06.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:06.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:14:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:07.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:14:07.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:07.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:07.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:07.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:07.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:07.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:07.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:07.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:07.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:14:07.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:07.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:07.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:07.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:07.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:07.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:08.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:08.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:08.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:14:08.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:08.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:14:08.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:08.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:08.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:08.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:14:08.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:08.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:08.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:08.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:14:08.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:14:08.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:14:08.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:14:09.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:14:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-15T10:14:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:14:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:14:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-15T10:14:09.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:14:09.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:14:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:14:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:14:09.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:14:10.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:14:10.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:10.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:14:10.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:14:10.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:14:10.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:14:10.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:14:10.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:14:12.443 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:14:12.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:14:12.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:14:12.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:14:12.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:14:12.992 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:14:12.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:14:12.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:14:12.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:14:12.995 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:14:12.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:14:12.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:14:12.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:14:12.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:14:12.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:14:12.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:14:13.030 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:13.031+0000 7f51b90407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:13.035+0000 7f51b90407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:13.034 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:13.035+0000 7f51b90407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:13.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:13.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:13.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:13.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:13.587+0000 7f51b90407c0 -1 Falling back to public interface 2024-06-15T10:14:14.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:14.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:14.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:14:14.471 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:14:14.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:14.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:14.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:14:15.729 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:14:15.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:15.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:15.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:16.542 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:16.543+0000 7f51b90407c0 -1 osd.1 48 log_to_monitors true 2024-06-15T10:14:16.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:16.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:16.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:14:16.971 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:14:16.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:16.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 53 up_thru 43 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/820044721,v1:127.0.0.1:6811/820044721] [v2:127.0.0.1:6812/820044721,v1:127.0.0.1:6813/820044721] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:14:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:14:17.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:14:17.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:14:17.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:14:17.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:14:17.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:14:17.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:14:17.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:14:18.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:14:18.270 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:14:18.270 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:14:18.270 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:14:18.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:14:18.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:18.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:14:18.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-06-15T10:14:18.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-06-15T10:14:18.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-06-15T10:14:18.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:18.617 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:14:18.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-06-15T10:14:18.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-06-15T10:14:18.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-227633266690' 2024-06-15T10:14:18.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:18.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:14:19.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-06-15T10:14:19.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-06-15T10:14:19.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-227633266690 2-64424509469' 2024-06-15T10:14:19.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:19.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:14:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495622 2024-06-15T10:14:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495622 2024-06-15T10:14:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-227633266690 2-64424509469 3-197568495622' 2024-06-15T10:14:19.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:19.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-06-15T10:14:19.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:19.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:14:19.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-06-15T10:14:19.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-06-15T10:14:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-06-15T10:14:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836529 2024-06-15T10:14:19.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:19.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-06-15T10:14:19.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:14:20.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:14:20.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:20.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836529 2024-06-15T10:14:20.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:20.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:20.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-06-15T10:14:20.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:14:20.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-06-15T10:14:20.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:20.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-06-15T10:14:20.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-06-15T10:14:20.748 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 227633266690 2024-06-15T10:14:20.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:14:21.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-06-15T10:14:21.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:21.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-06-15T10:14:21.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:21.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:14:21.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-06-15T10:14:21.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:21.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-06-15T10:14:21.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-06-15T10:14:21.054 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509469 2024-06-15T10:14:21.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:14:21.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-06-15T10:14:21.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:21.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495622 2024-06-15T10:14:21.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:21.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:14:21.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495622 2024-06-15T10:14:21.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:21.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495622 2024-06-15T10:14:21.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-06-15T10:14:21.291 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 197568495622 2024-06-15T10:14:21.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:14:21.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495622 2024-06-15T10:14:21.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:14:21.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:21.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:21.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:14:21.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:21.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:22.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:14:22.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:22.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:22.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:22.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:14:22.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:14:22.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:14:22.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:14:22.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:14:22.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:14:22.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:14:22.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:14:22.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:22.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:22.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:14:23.144 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:14:23.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:23.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:14:23.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:23.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:14:23.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:14:23.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:23.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:23.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:23.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:23.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:23.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:23.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:48.359637+0000 '>' 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:23.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:14:24.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:14:24.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:24.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:24.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:24.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:24.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:24.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:24.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:48.359637+0000 '>' 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:24.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:14:25.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:14:25.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:25.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:25.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:25.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:25.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:25.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:25.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:48.359637+0000 '>' 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:25.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:14:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:14:26.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:26.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:26.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:26.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:26.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:26.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:13:48.359637+0000 '>' 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:27.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:14:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:14:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:28.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:28.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:28.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:28.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:28.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:23.450396+0000 '>' 2024-06-15T10:13:48.359637+0000 2024-06-15T10:14:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:14:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:14:28.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:14:28.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:14:28.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:14:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:14:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:14:28.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:14:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:14:28.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:14:28.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:14:30.991 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:14:30.992 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:14:30.992 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:14:30.992 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:14:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:14:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:14:31.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:14:31.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:14:31.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:14:31.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:14:31.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:14:31.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:14:31.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:14:31.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:14:31.303 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:14:31.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:14:31.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:14:31.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:14:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:14:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:14:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:14:31.336 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:31.335+0000 7f112914a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:31.338 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:31.339+0000 7f112914a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:31.340 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:31.339+0000 7f112914a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:31.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:14:31.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:14:31.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:31.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:31.887 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:31.891+0000 7f112914a7c0 -1 Falling back to public interface 2024-06-15T10:14:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:14:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:14:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:33.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:14:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:14:34.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:34.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:34.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:34.726 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:14:34.731+0000 7f112914a7c0 -1 osd.3 55 log_to_monitors true 2024-06-15T10:14:35.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:35.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:35.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:14:35.371 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:14:35.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:35.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:14:35.996 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/3398782206,v1:127.0.0.1:6827/3398782206] [v2:127.0.0.1:6828/3398782206,v1:127.0.0.1:6829/3398782206] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:14:35.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:14:35.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:14:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:14:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:14:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:14:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:14:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:14:36.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:14:36.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:36.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:14:36.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-06-15T10:14:36.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-06-15T10:14:36.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-06-15T10:14:36.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:36.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:14:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-06-15T10:14:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-06-15T10:14:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-227633266694' 2024-06-15T10:14:36.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:36.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:14:37.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-06-15T10:14:37.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-06-15T10:14:37.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-227633266694 2-64424509473' 2024-06-15T10:14:37.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:37.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:14:37.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-06-15T10:14:37.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-06-15T10:14:37.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-227633266694 2-64424509473 3-253403070466' 2024-06-15T10:14:37.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:37.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-06-15T10:14:37.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:37.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:14:37.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-06-15T10:14:37.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:37.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-06-15T10:14:37.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-06-15T10:14:37.276 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836534 2024-06-15T10:14:37.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:37.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-06-15T10:14:37.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:14:38.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:14:38.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:38.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836534 2024-06-15T10:14:38.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:38.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-06-15T10:14:38.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:38.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:14:38.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-06-15T10:14:38.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:38.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-06-15T10:14:38.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-06-15T10:14:38.754 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 227633266694 2024-06-15T10:14:38.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:14:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266695 -lt 227633266694 2024-06-15T10:14:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:39.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-06-15T10:14:39.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:39.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:14:39.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:39.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-06-15T10:14:39.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-06-15T10:14:39.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-06-15T10:14:39.019 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509473 2024-06-15T10:14:39.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:14:39.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509473 2024-06-15T10:14:39.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:39.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-06-15T10:14:39.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:39.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:14:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-06-15T10:14:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-06-15T10:14:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-06-15T10:14:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 253403070466 2024-06-15T10:14:39.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:14:39.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-06-15T10:14:39.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:14:39.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:39.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:39.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:14:39.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:39.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:40.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:14:40.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:40.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:40.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:40.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:14:40.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:14:40.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:14:40.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:14:40.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-15T10:14:40.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162559"' 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162559' 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 162561"' 2024-06-15T10:14:40.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 162561' 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 162559 162561' 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162559 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162562: /' 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:14:40.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/162564: /' 2024-06-15T10:14:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:14:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:14:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:14:43.911 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:14:43.912 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:14:43.913 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:14:43.916 INFO:tasks.workunit.client.0.smithi116.stderr:162562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:14:43.917 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:14:43.918 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:14:43.918 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:14:43.918 INFO:tasks.workunit.client.0.smithi116.stderr:162562: start osd.1 2024-06-15T10:14:43.918 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:14:44.008 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:14:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:14:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:14:44.011 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:14:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:14:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:14:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:14:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:14:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:14:44.013 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:14:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:14:44.015 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:14:44.016 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:14:44.017 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:14:44.017 INFO:tasks.workunit.client.0.smithi116.stderr:162564: start osd.2 2024-06-15T10:14:44.017 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2024-06-15T10:14:44.047+0000 7f345e7d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2024-06-15T10:14:44.051+0000 7f345e7d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2024-06-15T10:14:44.051+0000 7f345e7d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:14:48.446 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 0 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2024-06-15T10:14:44.595+0000 7f345e7d77c0 -1 Falling back to public interface 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.447 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 1 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.448 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 3 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2024-06-15T10:14:48.314+0000 7f345e7d77c0 -1 osd.2 60 log_to_monitors true 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:14:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2024-06-15T10:14:43.959+0000 7fd25675b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2024-06-15T10:14:43.963+0000 7fd25675b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2024-06-15T10:14:43.963+0000 7fd25675b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 0 2024-06-15T10:14:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2024-06-15T10:14:44.503+0000 7fd25675b7c0 -1 Falling back to public interface 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 1 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2024-06-15T10:14:47.938+0000 7fd25675b7c0 -1 osd.1 60 log_to_monitors true 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 3 2024-06-15T10:14:48.452 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:48.453 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 4 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/2863613282,v1:127.0.0.1:6819/2863613282] [v2:127.0.0.1:6820/2863613282,v1:127.0.0.1:6821/2863613282] exists,up 25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:14:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:14:50.596 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 1 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 2 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: 3' 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-06-15T10:14:50.597 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 4 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: osd.1 up in weight 1 up_from 66 up_thru 66 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/2535981972,v1:127.0.0.1:6811/2535981972] [v2:127.0.0.1:6812/2535981972,v1:127.0.0.1:6813/2535981972] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:14:50.598 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:14:50.599 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 1 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 2 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: 3' 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-06-15T10:14:50.600 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841538' 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:14:52.890 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841538 2-287762808834' 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070471 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070471 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841538 2-287762808834 3-253403070471' 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:52.891 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: waiting osd.0 seq 21474836539 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836539 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836539 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:52.892 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-06-15T10:14:52.893 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:52.893 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:14:52.893 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-06-15T10:14:52.893 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/upers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-283467841539' 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-283467841539 2-287762808835' 2024-06-15T10:14:52.899 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-283467841539 2-287762808835 3-253403070470' 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-06-15T10:14:52.900 INFO:tasks.workunit.client.0.smithi116.stderr:162564: waiting osd.0 seq 21474836538 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836538 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836538 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-06-15T10:14:52.901 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: waiting osd.1 seq 283467841539 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:53.985 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: waiting osd.2 seq 287762808835 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:14:53.986 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: waiting osd.3 seq 253403070470 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:53.987 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: waiting osd.1 seq 283467841538 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:14:53.988 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: waiting osd.2 seq 287762808834 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070471 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070471 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:14:53.989 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070471 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: waiting osd.3 seq 253403070471 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070471 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:53.990 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:55.040 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:55.040 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:55.040 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:55.041 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:14:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /stale") | not)' 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:55.044 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:55.046 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:14:56.237 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:56.238 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:56.239 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:56.241 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:14:56.242 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:14:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:14:57.171 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:57.172 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr::1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:14:57.173 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:14:57.174 INFO:tasks.workunit.client.0.smithi116.stderr:162564: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 162561 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:14:57.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:14:57.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:14:57.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:14:57.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:14:57.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:14:57.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:14:57.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:14:57.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:57.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:57.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:57.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:57.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:14:23.450396+0000 2024-06-15T10:14:57.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:14:57.922 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:14:23.450396+0000 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:14:23.450396+0000 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:14:57.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:57.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:57.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:57.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:57.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:57.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:58.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:23.450396+0000 '>' 2024-06-15T10:14:23.450396+0000 2024-06-15T10:14:58.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:14:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:14:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:14:59.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:14:59.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:14:59.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:14:59.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:14:59.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:14:59.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:23.450396+0000 '>' 2024-06-15T10:14:23.450396+0000 2024-06-15T10:14:59.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:00.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:00.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:00.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:00.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:00.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:00.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:00.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:23.450396+0000 '>' 2024-06-15T10:14:23.450396+0000 2024-06-15T10:15:00.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:01.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:01.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:01.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:01.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:01.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:01.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:01.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:58.546100+0000 '>' 2024-06-15T10:14:23.450396+0000 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165827"' 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165827' 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:15:01.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165829: /' 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165830"' 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165830' 2024-06-15T10:15:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165827 165830' 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165827 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:15:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165832: /' 2024-06-15T10:15:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:15:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:05.247 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:15:05.248 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: _ 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: hinfo_key 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: snapset 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.249 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:15:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:15:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:05.250 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:05.252 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:05.252 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:05.252 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:05.253 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:15:05.254 INFO:tasks.workunit.client.0.smithi116.stderr:165832: start osd.2 2024-06-15T10:15:05.295 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.295 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:05.295 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:15:05.295 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: _ 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: hinfo_key 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: snapset 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:05.297 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:05.298 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:15:05.298 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:05.298 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:05.298 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:05.298 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:05.299 INFO:tasks.workunit.client.0.smithi116.stderr:165829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:05.300 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:05.301 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:05.301 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:15:05.301 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:15:05.301 INFO:tasks.workunit.client.0.smithi116.stderr:165829: start osd.1 2024-06-15T10:15:09.792 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:15:09.792 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:15:09.792 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:15:09.792 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2024-06-15T10:15:05.306+0000 7fc3d06407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2024-06-15T10:15:05.310+0000 7fc3d06407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2024-06-15T10:15:05.314+0000 7fc3d06407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.793 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 0 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2024-06-15T10:15:05.886+0000 7fc3d06407c0 -1 Falling back to public interface 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 1 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:15:09.794 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 3 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.795 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2024-06-15T10:15:09.310+0000 7fc3d06407c0 -1 osd.2 69 log_to_monitors true 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2024-06-15T10:15:05.334+0000 7f7faf0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.796 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2024-06-15T10:15:05.334+0000 7f7faf0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2024-06-15T10:15:05.338+0000 7f7faf0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 0 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2024-06-15T10:15:05.882+0000 7f7faf0997c0 -1 Falling back to public interface 2024-06-15T10:15:09.797 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 1 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.798 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2024-06-15T10:15:09.022+0000 7f7faf0997c0 -1 osd.1 70 log_to_monitors true 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 3 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:09.799 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 4 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: osd.2 up in weight 1 up_from 75 up_thru 49 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6818/3008150208,v1:127.0.0.1:6819/3008150208] [v2:127.0.0.1:6820/3008150208,v1:127.0.0.1:6821/3008150208] exists,up 25912c94-7f79-4981-a94a-a32e3e3aeeb2 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:12.320 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:12.321 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 1 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 2 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: 3' 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:12.322 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standaloneph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 4 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: osd.1 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [66,70) [v2:127.0.0.1:6810/2578618141,v1:127.0.0.1:6811/2578618141] [v2:127.0.0.1:6812/2578618141,v1:127.0.0.1:6813/2578618141] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:12.324 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 1 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 2 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: 3' 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-06-15T10:15:12.325 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-06-15T10:15:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:14.520 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:14.520 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-06-15T10:15:14.520 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-317827579907' 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-317827579907 2-322122547203' 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-317827579907 2-322122547203 3-253403070476' 2024-06-15T10:15:14.521 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: waiting osd.0 seq 21474836544 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836544 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:14.522 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836544 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-06-15T10:15:14.523 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:14.527 INFO:tasks.workunit.client.0.smithi116.stderr:16582e/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:14.527 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-15T10:15:14.527 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-15T10:15:14.527 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-317827579906' 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-317827579906 2-322122547202' 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-317827579906 2-322122547202 3-253403070477' 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-06-15T10:15:14.528 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: waiting osd.0 seq 21474836545 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836545 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836545 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:14.529 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579906 2024-06-15T10:15:14.530 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:14.530 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:15:14.530 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579906 2024-06-15T10:15:14.530 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:15.526 INFO:tasks.workunit.client.0.smithi116.stderr:165839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-06-15T10:15:15.526 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-06-15T10:15:15.526 INFO:tasks.workunit.client.0.smithi116.stderr:165829: waiting osd.1 seq 317827579907 2024-06-15T10:15:15.526 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547203 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547203 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547203' 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: waiting osd.2 seq 322122547203 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-06-15T10:15:15.527 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: waiting osd.3 seq 253403070476 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:15.528 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:15.529 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:15:15.529 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:15.529 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:15.529 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:15.529 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(2: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: waiting osd.1 seq 317827579906 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:15.538 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: waiting osd.2 seq 322122547202 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:15.539 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: waiting osd.3 seq 253403070477 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070477 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:16.653 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:15:16.654 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:16.655 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:165829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:16.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165830 2024-06-15T10:15:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:contains("stale") | not)' 2024-06-15T10:15:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:15:16.659 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:16.660 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:165832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:15:16.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:15:16.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:15:16.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:15:16.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:15:16.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:15:16.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:16.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168652"' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168652' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168655"' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168654: /' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168655' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 168652 168655' 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168652 2024-06-15T10:15:16.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:15:16.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168657: /' 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:15:20.260 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:15:20.261 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:20.262 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:20.263 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:20.263 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:20.265 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:20.266 INFO:tasks.workunit.client.0.smithi116.stderr:168657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:20.267 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:20.267 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:20.267 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:20.269 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:20.269 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: start osd.1 2024-06-15T10:15:20.270 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:15:21.292 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:21.292 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:21.292 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:15:21.292 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:21.292 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:15:21.293 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:21.294 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:15:21.295 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:21.295 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:21.298 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:21.299 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: start osd.3 2024-06-15T10:15:21.300 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:15:24.937 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:15:24.937 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2024-06-15T10:15:20.302+0000 7fdbc4a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2024-06-15T10:15:20.306+0000 7fdbc4a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2024-06-15T10:15:20.306+0000 7fdbc4a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:15:24.938 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 0 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2024-06-15T10:15:20.862+0000 7fdbc4a9d7c0 -1 Falling back to public interface 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 1 2024-06-15T10:15:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2024-06-15T10:15:24.301+0000 7fdbc4a9d7c0 -1 osd.1 78 log_to_monitors true 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:24.941 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:24.941 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 3 2024-06-15T10:15:24.941 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:24.941 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:25.761 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2024-06-15T10:15:21.334+0000 7f87477047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:25.762 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2024-06-15T10:15:21.338+0000 7f87477047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:25.762 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2024-06-15T10:15:21.338+0000 7f87477047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:25.762 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:15:25.762 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:25.762 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 0 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2024-06-15T10:15:21.902+0000 7f87477047c0 -1 Falling back to public interface 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 1 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2 2024-06-15T10:15:25.764 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2024-06-15T10:15:25.137+0000 7f87477047c0 -1 osd.3 78 log_to_monitors true 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 3 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:27.092 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 4 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: osd.1 up in weight 1 up_from 83 up_thru 75 down_at 79 last_clean_interval [74,78) [v2:127.0.0.1:6810/4211564668,v1:127.0.0.1:6811/4211564668] [v2:127.0.0.1:6812/4211564668,v1:127.0.0.1:6813/4211564668] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:27.093 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:27.094 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 1 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 2 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: 3' 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-06-15T10:15:27.095 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:28.164 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:28.164 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 4 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: osd.3 up in weight 1 up_from 86 up_thru 75 down_at 79 last_clean_interval [59,78) [v2:127.0.0.1:6826/1440030557,v1:127.0.0.1:6827/1440030557] [v2:127.0.0.1:6828/1440030557,v1:127.0.0.1:6829/1440030557] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:28.165 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:28.166 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 1 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 2 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: 3' 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-06-15T10:15:28.167 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:29.044 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-356482285570' 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547207 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547207 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-356482285570 2-322122547207' 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187458 2024-06-15T10:15:29.045 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187458 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-356482285570 2-322122547207 3-369367187458' 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: waiting osd.0 seq 21474836549 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:29.046 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: waiting osd.1 seq 356482285570 2024-06-15T10:15:29.047 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221pers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-356482285571' 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547208 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547208 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-356482285571 2-322122547208' 2024-06-15T10:15:30.444 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187459 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187459 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-356482285571 2-322122547208 3-369367187459' 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-06-15T10:15:30.445 INFO:tasks.workunit.client.0.smithi116.stderr:168654: waiting osd.0 seq 21474836550 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836550 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836550 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-06-15T10:15:30.446 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/u4: flush_pg_stats: test 356482285570 -lt 356482285570 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547207 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547207 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547207 2024-06-15T10:15:31.279 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547207' 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: waiting osd.2 seq 322122547207 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547207 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187458 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187458 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187458 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: waiting osd.3 seq 369367187458 2024-06-15T10:15:31.280 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 369367187458 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187458 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:31.281 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") |buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: waiting osd.1 seq 356482285571 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547208 2024-06-15T10:15:31.508 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547208 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547208 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547208' 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: waiting osd.2 seq 322122547208 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547208 -lt 322122547208 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187459 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:31.509 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187459 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187459 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: waiting osd.3 seq 369367187459 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187459 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:31.510 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:31.511 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:31.511 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:32.409 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains(" not)' 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:15:32.410 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:168657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:32.412 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:32.412 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:32.412 INFO:tasks.workunit.client.0.smithi116.stderr:168657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:32.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:stale") | not)' 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:32.613 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:168654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168655 2024-06-15T10:15:32.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:15:32.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:15:32.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:15:32.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:15:32.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:15:32.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:15:32.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:15:32.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:15:32.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:15:32.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:15:32.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:15:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:15:32.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:15:32.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:15:32.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:32.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:32.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:32.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:33.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:33.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:15:33.296 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:15:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:15:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:15:33.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:15:33.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:33.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:33.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:33.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:33.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:33.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:58.546100+0000 '>' 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:34.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:34.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:34.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:34.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:34.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:34.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:34.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:34.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:58.546100+0000 '>' 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:34.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:35.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:35.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:35.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:35.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:35.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:35.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:35.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:36.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:58.546100+0000 '>' 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:36.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:37.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:37.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:37.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:37.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:37.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:37.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:37.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:14:58.546100+0000 '>' 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:37.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:15:38.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:15:38.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:15:38.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:15:38.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:15:38.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:15:38.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:15:38.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:15:38.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:15:34.217578+0000 '>' 2024-06-15T10:14:58.546100+0000 2024-06-15T10:15:38.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:15:38.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171651"' 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171651' 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171653"' 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171653' 2024-06-15T10:15:38.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171654: /' 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171651 171653' 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171651 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:38.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171657: /' 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:41.832 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:15:41.833 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: _ 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: hinfo_key 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: snapset 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:15:41.834 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:41.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:41.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:15:41.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: start osd.3 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:15:42.412 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:15:42.413 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: _ 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: hinfo_key 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: snapset 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:15:42.414 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:15:42.417 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:42.418 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:15:42.419 INFO:tasks.workunit.client.0.smithi116.stderr:171657: start osd.1 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2024-06-15T10:15:41.873+0000 7f68fcd0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2024-06-15T10:15:41.877+0000 7f68fcd0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.047 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2024-06-15T10:15:41.877+0000 7f68fcd0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 0 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2024-06-15T10:15:42.421+0000 7f68fcd0a7c0 -1 Falling back to public interface 2024-06-15T10:15:46.048 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 1 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:46.049 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2024-06-15T10:15:45.501+0000 7f68fcd0a7c0 -1 osd.3 88 log_to_monitors true 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 3 2024-06-15T10:15:46.050 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.941 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2024-06-15T10:15:42.449+0000 7f8cdcdd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2024-06-15T10:15:42.493+0000 7f8cdcdd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2024-06-15T10:15:42.497+0000 7f8cdcdd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:15:46.942 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 0 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2024-06-15T10:15:43.029+0000 7f8cdcdd37c0 -1 Falling back to public interface 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 1 2024-06-15T10:15:46.943 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2024-06-15T10:15:46.469+0000 7f8cdcdd37c0 -1 osd.1 88 log_to_monitors true 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:46.944 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:15:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 3 2024-06-15T10:15:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:46.945 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 4 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:15:48.589 INFO:tasks.workunit.client.0.smithi116.stderr:171654: osd.3 up in weight 1 up_from 93 up_thru 86 down_at 89 last_clean_interval [86,88) [v2:127.0.0.1:6810/3190499083,v1:127.0.0.1:6811/3190499083] [v2:127.0.0.1:6812/3190499083,v1:127.0.0.1:6813/3190499083] exists,up bb180a4a-91ab-436e-b208-82a0e618b076 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:48.590 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 1 2024-06-15T10:15:48.591 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 2 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: 3' 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-06-15T10:15:48.592 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-06-15T10:15:50.840 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperseph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 4 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:15:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 5 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: osd.1 up in weight 1 up_from 97 up_thru 97 down_at 89 last_clean_interval [83,88) [v2:127.0.0.1:6826/3573137713,v1:127.0.0.1:6827/3573137713] [v2:127.0.0.1:6828/3573137713,v1:127.0.0.1:6829/3573137713] exists,up 143ad697-a0d4-467c-8fb2-1f0f68eb917e 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:15:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:171657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 1 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 2 2024-06-15T10:15:50.843 INFO:tasks.workunit.client.0.smithi116.stderr:171657: 3' 2024-06-15T10:15:50.844 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:15:50.844 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:17165.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: Error ENXIO: problem getting command descriptions from osd.1 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-06-15T10:15:51.475 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547213 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547213 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 2-322122547213' 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 2-322122547213 3-399431958530' 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: waiting osd.0 seq 21474836555 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836555 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836555 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547213 2024-06-15T10:15:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:51.478 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:51.478 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547213 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547213 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547213' 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: waiting osd.2 seq 322122547213 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547213 -lt 322122547213 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:52.805 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958530 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958530 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: waiting osd.3 seq 399431958530 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:52.806 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:52.808 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:52.808 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/7: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-416611827714' 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:15:53.077 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547215 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547215 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-416611827714 2-322122547215' 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958532 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958532 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-416611827714 2-322122547215 3-399431958532' 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: waiting osd.0 seq 21474836557 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836557 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:15:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.cceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:53.837 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:15:53.838 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:53.839 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clelient.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-416611827714 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-416611827714 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:54.137 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 416611827714' 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: waiting osd.1 seq 416611827714 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827714 -lt 416611827714 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547215 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547215 2024-06-15T10:15:54.138 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547215 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547215' 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: waiting osd.2 seq 322122547215 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547215 -lt 322122547215 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958532 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958532 2024-06-15T10:15:54.139 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958532 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958532' 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: waiting osd.3 seq 399431958532 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958532 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:54.140 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:54.821 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: testan: true 2024-06-15T10:15:54.821 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:54.821 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:54.821 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:15:54.822 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:55.362 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 5 == 0 2024-06-15T10:15:55.362 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:55.362 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:55.362 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:55.362 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:55.363 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:55.364 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:15:55.954 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_makin 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:15:55.955 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:55.956 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:55.957 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:56.206 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:56.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:15:56.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171653 2024-06-15T10:15:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:g_recovery_progress 2024-06-15T10:15:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:15:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:15:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:15:56.296 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:15:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:15:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:171657: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:15:56.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:15:56.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:56.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:56.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:56.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:15:56.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:15:56.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:15:56.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:15:56.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:15:56.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:15:56.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:15:56.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:15:56.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:15:56.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:15:56.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:15:56.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:15:56.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:15:56.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:15:56.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:56.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:56.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:15:56.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:15:56.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:15:56.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:15:56.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:15:56.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:15:56.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:15:56.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:15:56.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:15:56.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:15:56.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:15:56.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:15:56.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:15:56.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:15:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:15:56.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:15:56.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:15:56.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:15:56.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:56.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:56.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:15:56.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:15:56.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:15:56.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:15:56.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:15:56.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:56.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:56.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:15:56.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:15:56.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:15:56.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:15:56.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:15:56.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:15:56.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:15:56.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:15:57.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:15:57.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:57.150 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:57.150 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:57.151 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:57.151 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:57.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:57.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:15:57.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:15:57.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:15:57.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:15:57.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:15:57.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:15:57.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:15:57.388 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:15:57.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:15:57.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:15:57.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:15:57.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:15:57.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:15:57.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:15:57.503 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:15:57.503 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:15:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:15:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:15:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:57.504 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:15:57.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:15:57.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:15:57.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:15:58.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:15:58.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:58.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:58.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:58.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:15:58.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:15:58.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-06-15T10:15:58.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:15:58.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:15:58.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:15:58.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:15:58.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:15:58.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:15:58.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ecf9785f-22fb-47c7-b43d-2c7db564c0cb 2024-06-15T10:15:58.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ecf9785f-22fb-47c7-b43d-2c7db564c0cb' 2024-06-15T10:15:58.586 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 ecf9785f-22fb-47c7-b43d-2c7db564c0cb 2024-06-15T10:15:58.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:15:58.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeaW1m45I/JBAADrtiUqaJXxcQXWHqc4xoXA== 2024-06-15T10:15:58.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeaW1m45I/JBAADrtiUqaJXxcQXWHqc4xoXA=="}' 2024-06-15T10:15:58.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ecf9785f-22fb-47c7-b43d-2c7db564c0cb -i td/osd-scrub-repair/0/new.json 2024-06-15T10:15:58.872 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:15:58.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:15:58.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeaW1m45I/JBAADrtiUqaJXxcQXWHqc4xoXA== --osd-uuid ecf9785f-22fb-47c7-b43d-2c7db564c0cb 2024-06-15T10:15:58.921 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.920+0000 7ffa21ee07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:58.965 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.964+0000 7ffa21ee07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:58.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.968+0000 7ffa21ee07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:15:58.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.968+0000 7ffa21ee07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:15:58.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.968+0000 7ffa21ee07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:15:58.967 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:58.968+0000 7ffa21ee07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:15:59.081 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:15:59.080+0000 7ffa21ee07c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:16:06.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:16:06.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:16:06.945 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:16:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:16:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:16:07.223 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:16:07.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:16:07.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:16:07.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:16:07.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:16:07.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:16:07.255 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:07.256+0000 7f4b5408e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:07.304 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:07.304+0000 7f4b5408e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:07.306 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:07.308+0000 7f4b5408e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:07.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:16:07.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:16:07.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:16:07.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:07.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:07.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:07.884 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:07.884+0000 7f4b5408e7c0 -1 Falling back to public interface 2024-06-15T10:16:08.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:08.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:08.715 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:16:08.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:16:08.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:08.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:08.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:09.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:09.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:09.957 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:16:09.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:16:09.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:09.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:10.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:10.812+0000 7f4b5408e7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:16:11.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:11.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:11.198 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:16:11.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:16:11.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:11.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:11.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:12.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:12.726 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:16:12.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:16:12.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:12.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:13.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:14.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:14.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:14.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:16:14.025 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:16:14.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:14.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:14.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:15.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:15.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:15.265 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:16:15.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:16:15.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:15.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:15.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:16.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:16.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:16.581 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:16:16.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:16:16.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:16.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:16.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:17.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:17.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:17.991 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:16:17.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:16:17.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:17.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:18.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:16:19.235 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:16:19.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:19.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:19.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:20.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:20.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:20.477 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:16:20.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:16:20.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:20.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:20.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:21.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:21.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:21.891 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:16:21.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:16:21.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:21.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:22.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:23.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:23.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:23.134 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:16:23.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:16:23.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:23.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:23.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:24.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:24.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:24.586 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:16:24.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:16:24.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:24.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:24.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:25.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:25.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:25.948 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:16:25.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:16:25.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:25.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:26.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:27.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:27.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:27.318 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:16:27.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:16:27.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:27.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:27.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:28.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:28.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:28.565 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:16:28.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:16:28.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:28.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:28.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:29.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:29.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:29.829 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:16:29.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:16:29.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:29.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:30.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:31.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:31.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:31.504 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:16:31.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:16:31.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:31.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:32.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:33.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:33.139 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:16:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:16:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:33.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:34.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:34.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:34.501 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:16:34.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:16:34.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:34.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:34.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:35.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:35.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:35.740 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:16:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:16:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:36.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:37.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:37.153 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:16:37.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:37.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:16:37.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:37.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:37.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:38.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:38.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:38.563 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:16:38.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:16:38.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:38.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:39.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:40.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:40.243 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:16:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:16:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:40.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:40.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:41.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:41.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:41.719 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:16:41.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:16:41.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:41.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:41.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:42.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:42.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:42.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:16:42.995 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:16:42.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:42.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:16:43.238 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3413487207,v1:127.0.0.1:6803/3413487207] [v2:127.0.0.1:6804/3413487207,v1:127.0.0.1:6805/3413487207] exists,up ecf9785f-22fb-47c7-b43d-2c7db564c0cb 2024-06-15T10:16:43.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:16:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:16:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:16:43.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:16:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f2751b12-3044-4cf9-ab75-7d3581db1099' 2024-06-15T10:16:43.246 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:16:43.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:16:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALam1mWVsMEBAA6E83/d9rohICo/A4GR2Xuw== 2024-06-15T10:16:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALam1mWVsMEBAA6E83/d9rohICo/A4GR2Xuw=="}' 2024-06-15T10:16:43.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2751b12-3044-4cf9-ab75-7d3581db1099 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:16:43.555 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:16:43.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:16:43.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALam1mWVsMEBAA6E83/d9rohICo/A4GR2Xuw== --osd-uuid f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:16:43.607 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.607+0000 7f179b9057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:43.691 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.691+0000 7f179b9057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:43.692 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.691+0000 7f179b9057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:43.693 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.691+0000 7f179b9057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:16:43.693 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.691+0000 7f179b9057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:16:43.693 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.691+0000 7f179b9057c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:16:43.779 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:43.783+0000 7f179b9057c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:16:51.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:16:51.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:16:51.933 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:16:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:16:51.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:16:52.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:16:52.199 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:16:52.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:16:52.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:16:52.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:16:52.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:16:52.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:52.223+0000 7fb398be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:52.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:52.227+0000 7fb398be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:52.228 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:52.227+0000 7fb398be17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:52.476 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:16:52.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:16:52.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:52.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:52.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:52.776 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:52.775+0000 7fb398be17c0 -1 Falling back to public interface 2024-06-15T10:16:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:53.713 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:16:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:16:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:53.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:54.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:54.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:54.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:16:54.955 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:16:54.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:54.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:55.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:55.841 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:16:55.839+0000 7fb398be17c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:16:56.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:56.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:56.199 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:16:56.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:16:56.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:56.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:56.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:57.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:57.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:57.636 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:16:57.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:16:57.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:57.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:58.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:16:59.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:16:59.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:16:59.023 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:16:59.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:16:59.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:16:59.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:16:59.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:00.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:00.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:00.266 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:17:00.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:17:00.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:00.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:00.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:01.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:01.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:01.506 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:17:01.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:17:01.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:01.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:01.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:02.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:02.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:02.746 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:17:02.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:17:02.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:02.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:03.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:04.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:04.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:04.177 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:17:04.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:17:04.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:04.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:04.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:05.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:05.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:05.423 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:17:05.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:17:05.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:05.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:05.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:06.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:06.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:06.785 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:17:06.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:17:06.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:06.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:07.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:08.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:08.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:08.261 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:17:08.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:17:08.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:08.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:08.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:09.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:09.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:09.504 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:17:09.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:17:09.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:09.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:10.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:10.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:10.745 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:17:10.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:17:10.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:10.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:11.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:12.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:12.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:12.254 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:17:12.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:17:12.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:12.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:12.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:13.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:13.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:13.499 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:17:13.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:17:13.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:13.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:13.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:14.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:14.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:14.744 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:17:14.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:17:14.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:14.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:15.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:17:16.244 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:17:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:16.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:16.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:17.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:17.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:17.993 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:17:17.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:17:17.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:17.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:18.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:19.233 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:17:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:17:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:19.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:20.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:20.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:20.813 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:17:20.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:17:20.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:20.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:21.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:22.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:22.532 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:17:22.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:17:22.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:22.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:22.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:23.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:23.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:23.884 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:17:23.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:17:23.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:23.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:25.428 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:17:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:17:25.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:25.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:26.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:27.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:27.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:27.068 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:17:27.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:17:27.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:27.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:17:27.346 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3021852931,v1:127.0.0.1:6811/3021852931] [v2:127.0.0.1:6812/3021852931,v1:127.0.0.1:6813/3021852931] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:17:27.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:17:27.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:17:27.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:17:27.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:17:27.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:17:27.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:17:27.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d299a592-234c-441b-9dec-c8415fa94bfa' 2024-06-15T10:17:27.357 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:17:27.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:17:27.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3am1m5h5VFhAAHM6v5mkIRemU43YDPfMktw== 2024-06-15T10:17:27.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3am1m5h5VFhAAHM6v5mkIRemU43YDPfMktw=="}' 2024-06-15T10:17:27.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d299a592-234c-441b-9dec-c8415fa94bfa -i td/osd-scrub-repair/2/new.json 2024-06-15T10:17:27.689 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:17:27.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:17:27.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3am1m5h5VFhAAHM6v5mkIRemU43YDPfMktw== --osd-uuid d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:17:27.738 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.738+0000 7f2e33a387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:27.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.826+0000 7f2e33a387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:27.825 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.826+0000 7f2e33a387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:27.826 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.830+0000 7f2e33a387c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:17:27.826 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.830+0000 7f2e33a387c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:17:27.826 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.830+0000 7f2e33a387c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:17:27.919 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:27.922+0000 7f2e33a387c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:17:36.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:17:36.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:17:36.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:17:36.427 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:17:36.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:17:36.699 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:17:36.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:17:36.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:17:36.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:17:36.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:17:36.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:17:36.732 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:36.734+0000 7f3c7193d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:36.734 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:36.738+0000 7f3c7193d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:36.735 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:36.738+0000 7f3c7193d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:17:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:36.932 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:17:36.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:17:36.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:36.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:37.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:37.305 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:37.306+0000 7f3c7193d7c0 -1 Falling back to public interface 2024-06-15T10:17:38.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:38.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:38.207 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:17:38.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:17:38.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:38.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:38.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:39.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:39.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:39.449 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:17:39.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:17:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:39.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:40.354 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:17:40.354+0000 7f3c7193d7c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:17:40.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:40.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:40.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:17:40.754 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:17:40.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:40.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:41.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:42.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:42.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:42.332 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:17:42.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:17:42.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:42.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:42.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:43.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:43.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:43.666 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:17:43.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:17:43.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:43.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:44.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:45.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:45.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:45.038 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:17:45.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:17:45.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:45.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:45.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:46.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:46.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:46.568 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:17:46.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:17:46.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:46.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:46.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:47.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:47.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:47.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:17:47.812 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:17:47.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:47.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:48.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:49.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:49.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:49.342 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:17:49.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:17:49.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:49.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:49.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:50.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:50.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:50.606 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:17:50.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:17:50.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:50.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:50.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:51.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:51.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:51.848 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:17:51.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:17:51.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:51.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:52.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:53.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:53.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:53.230 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:17:53.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:17:53.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:53.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:53.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:54.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:54.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:54.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:17:54.640 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:17:54.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:54.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:54.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:55.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:55.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:55.878 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:17:55.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:17:55.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:55.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:56.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:57.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:57.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:57.199 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:17:57.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:17:57.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:57.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:57.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:58.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:58.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:58.577 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:17:58.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:17:58.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:58.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:17:58.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:17:59.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:17:59.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:17:59.818 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:17:59.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:17:59.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:17:59.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:00.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:01.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:01.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:01.296 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:18:01.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:18:01.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:01.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:02.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:03.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:03.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:03.010 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:18:03.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:18:03.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:03.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:03.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:04.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:04.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:04.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:18:04.701 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:18:04.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:04.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:05.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:06.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:06.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:06.397 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:18:06.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:18:06.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:06.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:06.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:07.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:07.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:07.640 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:18:07.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:18:07.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:07.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:08.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:09.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:09.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:09.151 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:18:09.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:18:09.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:09.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:09.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:10.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:10.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:10.804 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:18:10.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:18:10.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:10.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:11.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:12.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:12.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:12.273 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:18:12.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:18:12.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:12.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:18:12.572 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/675596927,v1:127.0.0.1:6819/675596927] [v2:127.0.0.1:6820/675596927,v1:127.0.0.1:6821/675596927] exists,up d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:18:12.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:18:12.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:18:12.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:18:12.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-06-15T10:18:12.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:18:12.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:18:12.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:18:12.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:18:12.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:18:12.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:18:12.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:18:12.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:18:12.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:18:12.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c5998d60-97a0-410f-a3a9-3a3c8d2fa588' 2024-06-15T10:18:12.580 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:18:12.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:18:12.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkam1m9c4KJBAAsiXAlZzyeh/NVb5JZ9ypvg== 2024-06-15T10:18:12.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkam1m9c4KJBAAsiXAlZzyeh/NVb5JZ9ypvg=="}' 2024-06-15T10:18:12.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5998d60-97a0-410f-a3a9-3a3c8d2fa588 -i td/osd-scrub-repair/3/new.json 2024-06-15T10:18:13.046 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:18:13.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-15T10:18:13.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkam1m9c4KJBAAsiXAlZzyeh/NVb5JZ9ypvg== --osd-uuid c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:18:13.101 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.101+0000 7efcde7e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:13.181 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.181+0000 7efcde7e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:13.183 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.185+0000 7efcde7e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:13.184 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.185+0000 7efcde7e97c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:18:13.184 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.185+0000 7efcde7e97c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:18:13.184 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.185+0000 7efcde7e97c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:18:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:13.269+0000 7efcde7e97c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-15T10:18:21.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-15T10:18:21.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:18:21.217 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T10:18:21.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T10:18:21.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:18:21.493 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T10:18:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T10:18:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:18:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:18:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:18:21.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:18:21.526 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:21.525+0000 7f467c0387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:21.570 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:21.569+0000 7f467c0387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:21.571 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:21.573+0000 7f467c0387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:18:21.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T10:18:21.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:18:21.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:18:21.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:18:21.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:18:21.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:21.721 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:18:21.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:18:21.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:21.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:21.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:22.165 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:22.165+0000 7f467c0387c0 -1 Falling back to public interface 2024-06-15T10:18:22.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:22.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:22.958 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:18:22.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:18:22.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:22.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:23.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:24.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:24.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:24.194 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:18:24.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:18:24.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:24.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:24.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:25.111 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:18:25.113+0000 7f467c0387c0 -1 osd.3 0 log_to_monitors true 2024-06-15T10:18:25.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:25.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:25.436 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:18:25.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:18:25.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:25.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:25.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:26.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:26.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:26.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:18:26.847 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:18:26.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:26.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:27.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:28.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:28.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:28.380 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:18:28.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:18:28.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:28.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:28.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:29.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:29.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:29.625 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:18:29.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:18:29.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:29.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:30.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:31.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:31.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:31.010 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:18:31.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:18:31.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:31.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:31.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:32.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:32.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:32.463 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:18:32.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:18:32.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:32.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:32.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:33.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:33.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:33.708 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:18:33.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:18:33.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:33.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:35.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:35.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:35.046 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:18:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:18:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:35.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:35.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:36.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:36.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:36.414 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:18:36.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:18:36.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:36.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:36.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:37.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:37.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:37.656 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:18:37.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:18:37.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:37.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:38.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:39.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:39.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:39.171 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:18:39.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:18:39.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:39.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:39.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:40.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:40.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:40.459 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:18:40.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:18:40.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:40.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:40.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:41.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:41.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:41.704 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:18:41.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:18:41.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:41.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:42.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:43.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:43.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:43.194 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:18:43.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:18:43.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:43.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:44.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:45.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:45.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:45.060 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:18:45.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:18:45.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:45.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:45.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:46.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:46.417 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:18:46.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:18:46.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:46.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:47.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:48.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:48.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:48.052 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:18:48.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:18:48.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:48.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:48.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:49.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:49.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:49.489 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:18:49.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:18:49.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:49.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:49.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:50.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:50.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:50.916 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:18:50.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:18:50.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:50.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:51.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:52.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:52.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:52.725 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:18:52.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:18:52.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:52.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:54.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:54.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:54.083 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:18:54.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:18:54.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:54.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:18:55.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:18:55.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:18:55.866 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:18:55.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:18:55.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:18:55.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:18:56.203 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1327472075,v1:127.0.0.1:6827/1327472075] [v2:127.0.0.1:6828/1327472075,v1:127.0.0.1:6829/1327472075] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:18:56.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:18:56.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:18:56.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:18:56.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-06-15T10:18:56.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:18:56.473 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:18:56.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:18:56.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:18:57.044 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:18:57.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:18:58.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:19:01.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-06-15T10:19:01.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:19:01.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:19:01.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:19:01.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:19:01.272 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:19:01.272 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:19:01.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:19:01.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:19:01.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:19:01.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:19:01.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:01.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:19:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-06-15T10:19:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-06-15T10:19:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-06-15T10:19:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:01.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:19:01.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-06-15T10:19:01.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-06-15T10:19:01.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982' 2024-06-15T10:19:01.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:01.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:19:02.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-06-15T10:19:02.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-06-15T10:19:02.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982 2-64424509462' 2024-06-15T10:19:02.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:02.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:19:02.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-06-15T10:19:02.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-06-15T10:19:02.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672982 2-64424509462 3-85899345932' 2024-06-15T10:19:02.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:02.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-06-15T10:19:02.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:02.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:19:02.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-06-15T10:19:02.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:02.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-06-15T10:19:02.205 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836516 2024-06-15T10:19:02.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-06-15T10:19:02.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:02.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836516 2024-06-15T10:19:02.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:19:03.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:19:03.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:03.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836516 2024-06-15T10:19:03.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:03.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-06-15T10:19:03.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:03.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:19:03.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-06-15T10:19:03.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:03.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-06-15T10:19:03.673 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672982 2024-06-15T10:19:03.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-06-15T10:19:03.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:19:03.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672982 2024-06-15T10:19:03.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:03.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-06-15T10:19:03.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:03.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:19:03.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-06-15T10:19:03.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:03.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-06-15T10:19:03.911 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509462 2024-06-15T10:19:03.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-06-15T10:19:03.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:19:04.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-06-15T10:19:04.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:04.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-06-15T10:19:04.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:04.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:19:04.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-06-15T10:19:04.145 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:04.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-06-15T10:19:04.146 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345932 2024-06-15T10:19:04.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-06-15T10:19:04.147 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:19:04.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-06-15T10:19:04.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:19:04.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:04.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:04.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:19:04.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:04.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:05.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:19:05.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:05.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:05.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:05.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:19:05.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:19:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-06-15T10:19:05.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:19:05.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:19:06.683 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:19:06.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:19:07.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-15T10:19:07.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-15T10:19:08.346 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 allow_ec_overwrites to true 2024-06-15T10:19:08.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:19:08.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:19:08.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:19:08.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:19:08.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:19:08.372 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:19:08.372 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:19:08.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:19:08.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:19:08.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:19:08.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:19:08.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:19:08.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:19:08.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:19:08.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:19:08.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:19:08.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:08.762 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:19:09.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-06-15T10:19:09.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-06-15T10:19:09.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-06-15T10:19:09.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:09.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:19:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-06-15T10:19:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-06-15T10:19:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672985' 2024-06-15T10:19:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:09.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:19:09.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-15T10:19:09.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-15T10:19:09.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672985 2-64424509464' 2024-06-15T10:19:09.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:09.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:19:09.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345934 2024-06-15T10:19:09.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345934 2024-06-15T10:19:09.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672985 2-64424509464 3-85899345934' 2024-06-15T10:19:09.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:09.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-06-15T10:19:09.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:09.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:19:09.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-06-15T10:19:09.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:09.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-06-15T10:19:09.575 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836519 2024-06-15T10:19:09.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-06-15T10:19:09.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:09.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-06-15T10:19:09.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:19:10.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:19:10.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:11.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-06-15T10:19:11.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:11.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-06-15T10:19:11.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:11.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:19:11.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-06-15T10:19:11.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:11.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-06-15T10:19:11.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-06-15T10:19:11.163 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672985 2024-06-15T10:19:11.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:19:11.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-06-15T10:19:11.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:11.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-15T10:19:11.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:11.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:19:11.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-15T10:19:11.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:11.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-15T10:19:11.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-15T10:19:11.443 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509464 2024-06-15T10:19:11.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:19:11.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-15T10:19:11.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:11.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345934 2024-06-15T10:19:11.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:11.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:19:11.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345934 2024-06-15T10:19:11.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:11.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345934 2024-06-15T10:19:11.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-06-15T10:19:11.710 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345934 2024-06-15T10:19:11.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:19:11.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345934 -lt 85899345934 2024-06-15T10:19:11.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:19:11.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:11.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:12.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:19:12.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:12.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:12.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:19:12.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:12.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:12.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:19:12.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:19:12.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T10:19:12.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:19:12.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:19:12.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:19:13.142 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:19:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:19:13.560 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:19:13.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:19:13.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:19:13.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:19:14.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-15T10:19:14.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:19:14.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:19:14.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:19:14.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:19:14.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-15T10:19:14.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-15T10:19:14.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-15T10:19:14.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T10:19:14.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-15T10:19:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:19:14.427 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:19:14.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-15T10:19:14.692 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:14.692 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:14.692 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:19:14.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:19:14.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:19:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:19:14.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:19:16.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:19:17.979 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:19:18.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:19:18.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:19:18.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:19:18.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:19:18.518 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:19:18.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:19:18.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:19:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:19:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:19:18.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:19:18.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:19:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:18.552+0000 7f14165ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:18.554 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:18.552+0000 7f14165ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:18.556 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:18.556+0000 7f14165ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:18.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:19:18.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:18.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:19.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:19.121 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:19.120+0000 7f14165ee7c0 -1 Falling back to public interface 2024-06-15T10:19:20.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:19:20.075 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:20.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:20.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:21.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:21.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:21.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:19:21.316 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:21.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:21.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:21.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:22.077 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:22.056+0000 7f14165ee7c0 -1 osd.3 36 log_to_monitors true 2024-06-15T10:19:22.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:19:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:19:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 40 up_thru 31 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/3731741794,v1:127.0.0.1:6827/3731741794] [v2:127.0.0.1:6828/3731741794,v1:127.0.0.1:6829/3731741794] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:19:22.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:19:22.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:19:23.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:19:23.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:19:23.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:19:23.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:19:23.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:19:23.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:19:23.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:23.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:19:23.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-06-15T10:19:23.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-06-15T10:19:23.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-06-15T10:19:23.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:23.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:19:23.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-06-15T10:19:23.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-06-15T10:19:23.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949672989' 2024-06-15T10:19:23.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:23.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:19:24.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-06-15T10:19:24.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-06-15T10:19:24.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949672989 2-64424509468' 2024-06-15T10:19:24.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:24.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:19:24.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T10:19:24.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T10:19:24.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949672989 2-64424509468 3-171798691842' 2024-06-15T10:19:24.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:24.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-06-15T10:19:24.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:24.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:19:24.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-06-15T10:19:24.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:24.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-06-15T10:19:24.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-06-15T10:19:24.614 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836523 2024-06-15T10:19:24.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:24.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836523 2024-06-15T10:19:24.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:19:25.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:19:25.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:26.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-06-15T10:19:26.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:26.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-06-15T10:19:26.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:26.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:19:26.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-06-15T10:19:26.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:26.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-06-15T10:19:26.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-06-15T10:19:26.217 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672989 2024-06-15T10:19:26.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:19:26.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-06-15T10:19:26.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:26.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-06-15T10:19:26.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:26.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:19:26.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-06-15T10:19:26.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:26.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-06-15T10:19:26.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-06-15T10:19:26.510 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509468 2024-06-15T10:19:26.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:19:26.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-06-15T10:19:26.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:26.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-06-15T10:19:26.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:19:26.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-06-15T10:19:26.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T10:19:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-06-15T10:19:26.753 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 171798691842 2024-06-15T10:19:26.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:19:27.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-15T10:19:27.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:19:27.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:27.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:19:27.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:27.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:27.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:27.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:27.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:27.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:27.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:27.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:19:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:27.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:27.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:19:27.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:19:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:19:27.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:27.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:27.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:27.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:27.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:27.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:27.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:28.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:19:28.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:28.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:28.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:19:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:19:28.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:19:28.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:19:28.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:19:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:19:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:19:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:19:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:19:28.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:19:28.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:19:28.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:28.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:28.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:28.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:28.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:28.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:28.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:29.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:19:29.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:29.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:29.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:29.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:19:29.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:19:29.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:19:29.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:19:29.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:19:29.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:19:29.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:19:29.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:19:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:19:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:19:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:19:29.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:19:29.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:19:29.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:19:29.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:19:29.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:19:29.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:29.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:19:30.160 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:19:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:19:30.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:30.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:19:30.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:19:30.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:19:30.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:19:30.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:19:30.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:19:30.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:19:30.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:19:30.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:06.602765+0000 '>' 2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:30.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:19:31.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:19:31.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:19:31.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:19:31.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:19:31.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:19:31.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:19:31.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:19:31.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:06.602765+0000 '>' 2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:31.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:19:32.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:19:32.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:19:32.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:19:32.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:19:32.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:19:32.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:19:32.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:19:32.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:30.195694+0000 '>' 2024-06-15T10:19:06.602765+0000 2024-06-15T10:19:32.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:19:32.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:19:32.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:19:32.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:19:32.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:19:33.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:19:35.490 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:19:35.490 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:19:35.490 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:19:35.490 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:19:35.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:19:35.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:19:35.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:19:35.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:19:35.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:19:35.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:19:35.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:19:35.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:19:35.797 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:19:35.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:19:35.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:19:35.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:19:35.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:19:35.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:19:35.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:19:35.831 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:35.832+0000 7ff0348f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:35.834 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:35.832+0000 7ff0348f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:35.835 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:35.836+0000 7ff0348f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:19:36.041 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:19:36.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:36.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:36.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:36.384 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:36.384+0000 7ff0348f27c0 -1 Falling back to public interface 2024-06-15T10:19:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:19:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:37.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:38.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:38.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:38.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:19:38.567 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:38.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:38.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:38.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:39.245 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:39.247+0000 7ff0348f27c0 -1 osd.3 42 log_to_monitors true 2024-06-15T10:19:39.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:39.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:39.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:19:39.808 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:19:39.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:39.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 46 up_thru 40 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/904650821,v1:127.0.0.1:6827/904650821] [v2:127.0.0.1:6828/904650821,v1:127.0.0.1:6829/904650821] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:19:40.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:19:40.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:19:40.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:19:40.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:19:40.367 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:19:40.368 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:19:40.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:19:40.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:19:40.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:19:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:19:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:19:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:19:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:19:40.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:19:40.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:19:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:19:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:40.927 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:19:40.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:19:40.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:40.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:19:41.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-06-15T10:19:41.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-06-15T10:19:41.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-06-15T10:19:41.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:41.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:19:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672993 2024-06-15T10:19:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672993 2024-06-15T10:19:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949672993' 2024-06-15T10:19:41.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:41.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:19:41.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-06-15T10:19:41.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-06-15T10:19:41.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949672993 2-64424509472' 2024-06-15T10:19:41.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:41.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:19:41.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-06-15T10:19:41.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-06-15T10:19:41.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949672993 2-64424509472 3-197568495618' 2024-06-15T10:19:41.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:41.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-06-15T10:19:41.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:41.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:19:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-06-15T10:19:41.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:41.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-06-15T10:19:41.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-06-15T10:19:41.792 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836527 2024-06-15T10:19:41.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:42.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-06-15T10:19:42.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:19:43.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:19:43.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:43.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836527 2024-06-15T10:19:43.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:43.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672993 2024-06-15T10:19:43.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:43.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:19:43.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672993 2024-06-15T10:19:43.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:43.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672993 2024-06-15T10:19:43.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-06-15T10:19:43.466 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672993 2024-06-15T10:19:43.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:19:43.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672994 -lt 42949672993 2024-06-15T10:19:43.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:43.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-06-15T10:19:43.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:43.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:19:43.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-06-15T10:19:43.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:43.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-06-15T10:19:43.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-06-15T10:19:43.732 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509472 2024-06-15T10:19:43.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:19:44.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509472 2024-06-15T10:19:44.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:44.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-06-15T10:19:44.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:44.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:19:44.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-06-15T10:19:44.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-06-15T10:19:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-06-15T10:19:44.014 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 197568495618 2024-06-15T10:19:44.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:19:44.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-06-15T10:19:44.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:19:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:44.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:19:44.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:44.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:44.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:44.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:44.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:44.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:44.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:44.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:19:44.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:44.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:44.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:19:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:19:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:19:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:19:45.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:19:45.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:19:45.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:19:45.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:19:45.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:19:45.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:19:45.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:19:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:19:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:19:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:19:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:19:45.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:19:48.202 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:19:48.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:19:48.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:19:48.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:19:48.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:19:48.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:19:48.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:19:48.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:19:48.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:19:48.742 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:19:48.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:19:48.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:19:48.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:19:48.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:19:48.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:19:48.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:19:48.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:48.779+0000 7f2e2c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:48.784 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:48.783+0000 7f2e2c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:48.785 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:48.783+0000 7f2e2c14d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:48.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:19:49.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:49.322 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:49.319+0000 7f2e2c14d7c0 -1 Falling back to public interface 2024-06-15T10:19:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:19:50.207 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:19:50.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:19:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:19:51.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:19:52.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:19:52.223+0000 7f2e2c14d7c0 -1 osd.1 48 log_to_monitors true 2024-06-15T10:19:52.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:19:52.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:19:52.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:19:52.683 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:19:52.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:19:52.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 53 up_thru 43 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/1823662225,v1:127.0.0.1:6811/1823662225] [v2:127.0.0.1:6812/1823662225,v1:127.0.0.1:6813/1823662225] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:19:53.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:19:53.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:19:53.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:19:53.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:19:53.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:19:53.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:19:53.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:19:53.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:19:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:19:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:19:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:19:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:19:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:19:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:53.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:19:54.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-06-15T10:19:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-06-15T10:19:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-06-15T10:19:54.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:54.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:19:54.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-06-15T10:19:54.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-06-15T10:19:54.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-227633266690' 2024-06-15T10:19:54.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:54.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:19:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509476 2024-06-15T10:19:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509476 2024-06-15T10:19:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-227633266690 2-64424509476' 2024-06-15T10:19:54.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:19:54.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:19:54.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495622 2024-06-15T10:19:54.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495622 2024-06-15T10:19:54.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-227633266690 2-64424509476 3-197568495622' 2024-06-15T10:19:54.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:54.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-06-15T10:19:54.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:54.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:19:54.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-06-15T10:19:54.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:54.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-06-15T10:19:54.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-06-15T10:19:54.751 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836531 2024-06-15T10:19:54.751 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:55.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836531 2024-06-15T10:19:55.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:19:56.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:19:56.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:19:56.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-06-15T10:19:56.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:56.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:56.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-06-15T10:19:56.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:19:56.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-06-15T10:19:56.358 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:56.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-06-15T10:19:56.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-06-15T10:19:56.359 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 227633266690 2024-06-15T10:19:56.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:19:56.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-06-15T10:19:56.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:56.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509476 2024-06-15T10:19:56.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:56.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:19:56.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509476 2024-06-15T10:19:56.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509476 2024-06-15T10:19:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-06-15T10:19:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509476 2024-06-15T10:19:56.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:19:56.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509476 -lt 64424509476 2024-06-15T10:19:56.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:19:56.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495622 2024-06-15T10:19:56.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:19:56.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:19:56.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495622 2024-06-15T10:19:56.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:19:56.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495622 2024-06-15T10:19:56.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495622' 2024-06-15T10:19:56.901 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 197568495622 2024-06-15T10:19:56.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:19:57.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495622 2024-06-15T10:19:57.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:19:57.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:57.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:57.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:19:57.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:57.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:57.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:19:57.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:57.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:57.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:19:57.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:19:58.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:19:58.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:58.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:58.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:19:58.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:58.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:58.260 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-06-15T10:19:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:19:58.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:19:58.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:19:58.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:19:58.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:19:58.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:19:58.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:19:58.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:19:59.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:19:59.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:19:59.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:19:59.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:19:59.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:19:59.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:19:59.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:19:59.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:19:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:19:59.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:19:59.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:19:59.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:19:59.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:19:59.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:19:59.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:19:59.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:19:59.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:19:59.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:19:59.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:19:59.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:19:59.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:00.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:00.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:20:00.258 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:20:00.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:00.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:00.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:00.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:00.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:00.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:00.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:30.195694+0000 '>' 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:00.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:01.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:01.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:01.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:01.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:01.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:01.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:01.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:01.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:30.195694+0000 '>' 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:01.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:02.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:02.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:02.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:02.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:02.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:02.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:02.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:03.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:30.195694+0000 '>' 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:03.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:04.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:04.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:04.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:04.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:04.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:04.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:19:30.195694+0000 '>' 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:04.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:05.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:05.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:05.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:05.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:05.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:05.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:05.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:00.424158+0000 '>' 2024-06-15T10:19:30.195694+0000 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:05.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:05.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:05.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:05.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:05.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:20:05.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:20:08.101 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:20:08.102 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:20:08.102 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:20:08.102 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:20:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:20:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:08.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:08.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:08.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:08.398 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:08.398 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:08.398 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:08.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:08.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:20:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:20:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:20:08.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:20:08.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:20:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:20:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:20:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:20:08.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:20:08.433 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:20:08.431+0000 7f05b34067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:08.436 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:20:08.435+0000 7f05b34067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:08.438 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:20:08.435+0000 7f05b34067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:20:08.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:20:08.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:08.974 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:20:08.971+0000 7f05b34067c0 -1 Falling back to public interface 2024-06-15T10:20:09.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:09.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:09.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:20:09.880 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:20:09.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:09.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:20:10.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:11.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:20:11.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:11.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:20:11.583+0000 7f05b34067c0 -1 osd.3 55 log_to_monitors true 2024-06-15T10:20:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:20:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:20:12.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:12.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:20:12.865 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/2620416709,v1:127.0.0.1:6827/2620416709] [v2:127.0.0.1:6828/2620416709,v1:127.0.0.1:6829/2620416709] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:20:12.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:20:12.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:20:12.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:20:12.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:20:12.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:20:12.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:20:12.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:20:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:20:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:20:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:20:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:20:12.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:20:12.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:20:13.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:3' 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:13.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:20:13.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-06-15T10:20:13.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-06-15T10:20:13.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-06-15T10:20:13.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:13.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:20:13.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266695 2024-06-15T10:20:13.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266695 2024-06-15T10:20:13.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-227633266695' 2024-06-15T10:20:13.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:13.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:20:14.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509481 2024-06-15T10:20:14.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509481 2024-06-15T10:20:14.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-227633266695 2-64424509481' 2024-06-15T10:20:14.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:14.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:20:14.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-06-15T10:20:14.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-06-15T10:20:14.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-227633266695 2-64424509481 3-253403070466' 2024-06-15T10:20:14.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:14.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-06-15T10:20:14.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:14.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:20:14.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-06-15T10:20:14.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:14.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-06-15T10:20:14.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-06-15T10:20:14.204 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836536 2024-06-15T10:20:14.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:14.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836536 2024-06-15T10:20:14.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:20:15.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:20:15.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:15.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-06-15T10:20:15.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:15.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266695 2024-06-15T10:20:15.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:15.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:20:15.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266695 2024-06-15T10:20:15.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:15.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266695 2024-06-15T10:20:15.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266695' 2024-06-15T10:20:15.671 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 227633266695 2024-06-15T10:20:15.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:20:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266695 -lt 227633266695 2024-06-15T10:20:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:15.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509481 2024-06-15T10:20:15.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:15.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:20:15.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:15.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509481 2024-06-15T10:20:15.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509481 2024-06-15T10:20:15.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-06-15T10:20:15.923 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509481 2024-06-15T10:20:15.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:20:16.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509481 -lt 64424509481 2024-06-15T10:20:16.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:16.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-06-15T10:20:16.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:16.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:20:16.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-06-15T10:20:16.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:16.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-06-15T10:20:16.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-06-15T10:20:16.139 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 253403070466 2024-06-15T10:20:16.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:20:16.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-06-15T10:20:16.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:20:16.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:16.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:16.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:20:16.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:16.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:16.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:16.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:16.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:16.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:16.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:16.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:20:16.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:16.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:16.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:17.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:20:17.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:20:17.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:20:17.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:20:17.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:20:17.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-06-15T10:20:17.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189569"' 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189569' 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:17.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189571"' 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189571' 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189572: /' 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 189569 189571' 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189569 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:20:17.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189574: /' 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:20:20.502 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:20.503 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:20.504 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: start osd.1 2024-06-15T10:20:20.509 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:20.720 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-06-15T10:20:20.721 INFO:tasks.workunit.client.0.smithi116.stderr:189574: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:20.722 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:20.724 INFO:tasks.workunit.client.0.smithi116.stderr:189574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:20:20.725 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:20:20.726 INFO:tasks.workunit.client.0.smithi116.stderr:189574: start osd.2 2024-06-15T10:20:20.726 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2024-06-15T10:20:20.543+0000 7fc92576f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2024-06-15T10:20:20.547+0000 7fc92576f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:24.999 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2024-06-15T10:20:20.547+0000 7fc92576f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 0 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.000 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2024-06-15T10:20:21.099+0000 7fc92576f7c0 -1 Falling back to public interface 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 1 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.001 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2024-06-15T10:20:24.391+0000 7fc92576f7c0 -1 osd.1 60 log_to_monitors true 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 3 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.002 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:25.185 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:20:25.185 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:20:25.185 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2024-06-15T10:20:20.759+0000 7fecafbb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2024-06-15T10:20:20.763+0000 7fecafbb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2024-06-15T10:20:20.763+0000 7fecafbb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:25.186 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.187 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 0 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2024-06-15T10:20:21.315+0000 7fecafbb07c0 -1 Falling back to public interface 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.188 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 1 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:25.189 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2024-06-15T10:20:24.607+0000 7fecafbb07c0 -1 osd.2 60 log_to_monitors true 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 3 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:25.190 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 4 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: osd.1 up in weight 1 up_from 66 up_thru 66 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/112749393,v1:127.0.0.1:6811/112749393] [v2:127.0.0.1:6812/112749393,v1:127.0.0.1:6813/112749393] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:20:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:20:27.278 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 1 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 2 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: 3' 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-06-15T10:20:27.279 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperss.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 4 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:27.280 INFO:tasks.workunit.client.0.smithi116.stderr:189574: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/2773107571,v1:127.0.0.1:6819/2773107571] [v2:127.0.0.1:6820/2773107571,v1:127.0.0.1:6821/2773107571] exists,up d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:20:27.281 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 1 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 2 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: 3' 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:20:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-06-15T10:20:27.283 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-06-15T10:20:27.283 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-06-15T10:20:27.283 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:29.695 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841539' 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841539 2-287762808834' 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-06-15T10:20:29.696 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-283467841539 2-287762808834 3-253403070470' 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: waiting osd.0 seq 21474836539 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-06-15T10:20:29.697 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836539 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-06-15T10:20:29.698 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-283467841538' 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-283467841538 2-287762808835' 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070471 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070471 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-283467841538 2-287762808835 3-253403070471' 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: waiting osd.0 seq 21474836540 2024-06-15T10:20:29.700 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836540 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836540 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-06-15T10:20:29.701 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:30.704 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/uu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-06-15T10:20:30.704 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-06-15T10:20:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:189572: waiting osd.1 seq 283467841539 2024-06-15T10:20:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:20:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-06-15T10:20:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-06-15T10:20:30.706 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: waiting osd.2 seq 287762808834 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-06-15T10:20:30.707 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-06-15T10:20:30.708 INFO:tasks.workunit.client.0.smithi116.stderr:189572: waiting osd.3 seq 253403070470 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:30.709 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stalbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-06-15T10:20:30.710 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: waiting osd.1 seq 283467841538 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-06-15T10:20:30.711 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: waiting osd.2 seq 287762808835 2024-06-15T10:20:30.712 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070471 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070471 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070471 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: waiting osd.3 seq 253403070471 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070471 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:20:30.713 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:30.714 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:31.756 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("e") | not)' 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:20:31.757 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:31.758 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:20:31.759 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //homstale") | not)' 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:20:31.760 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:31.761 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:20:31.762 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:20:32.953 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:32.954 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:20:32.955 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:20:33.003 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:33.004 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:20:33.005 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh0: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:33.870 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:189572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:20:33.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189571 2024-06-15T10:20:33.921 INFO:tasks.workunit.client.0.smithi116.stderr::1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:20:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:20:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:20:33.921 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:189574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:20:33.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:20:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:20:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:20:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:20:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:20:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:20:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:20:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:20:34.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:20:34.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:20:34.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:34.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:34.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:34.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:34.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:34.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:20:34.634 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:20:34.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:34.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:34.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:34.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:34.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:34.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:34.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:00.424158+0000 '>' 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:34.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:35.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:35.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:35.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:36.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:00.424158+0000 '>' 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:36.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:37.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:37.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:37.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:37.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:37.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:37.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:37.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:37.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:00.424158+0000 '>' 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:37.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:38.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:38.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:38.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:38.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:38.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:00.424158+0000 '>' 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:38.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:20:39.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:20:39.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:20:39.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:20:39.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:20:39.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:20:39.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:20:39.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:34.727829+0000 '>' 2024-06-15T10:20:00.424158+0000 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 192859"' 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 192859' 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:20:39.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/192861: /' 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 192862"' 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 192862' 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 192859 192862' 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 192859 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:20:39.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/192865: /' 2024-06-15T10:20:43.869 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:20:43.870 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:20:43.871 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: _ 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: hinfo_key 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: snapset 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:20:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:43.873 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:43.874 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:43.874 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:43.875 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:43.876 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:20:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:20:43.877 INFO:tasks.workunit.client.0.smithi116.stderr:192865: start osd.2 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: _ 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: hinfo_key 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: snapset 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:43.962 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:43.963 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:20:43.964 INFO:tasks.workunit.client.0.smithi116.stderr:192861: start osd.1 2024-06-15T10:20:48.379 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2024-06-15T10:20:43.914+0000 7f45b4bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2024-06-15T10:20:43.918+0000 7f45b4bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2024-06-15T10:20:43.918+0000 7f45b4bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:20:48.380 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 0 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2024-06-15T10:20:44.458+0000 7f45b4bf77c0 -1 Falling back to public interface 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.381 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 1 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.382 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2024-06-15T10:20:47.450+0000 7f45b4bf77c0 -1 osd.2 69 log_to_monitors true 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 3 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-15T10:20:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2024-06-15T10:20:43.994+0000 7ff9303f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2024-06-15T10:20:43.998+0000 7ff9303f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2024-06-15T10:20:44.002+0000 7ff9303f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:20:48.384 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 0 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2024-06-15T10:20:44.550+0000 7ff9303f97c0 -1 Falling back to public interface 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 1 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.385 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:48.386 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:20:48.387 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 3 2024-06-15T10:20:48.387 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:48.387 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:48.387 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2024-06-15T10:20:48.002+0000 7ff9303f97c0 -1 osd.1 69 log_to_monitors true 2024-06-15T10:20:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:20:50.940 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 4 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: osd.1 up in weight 1 up_from 74 up_thru 67 down_at 70 last_clean_interval [66,69) [v2:127.0.0.1:6818/39482938,v1:127.0.0.1:6819/39482938] [v2:127.0.0.1:6820/39482938,v1:127.0.0.1:6821/39482938] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:20:50.941 INFO:tasks.workunit.client.0.smithi116.stderr:192861: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 1 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 2 2024-06-15T10:20:50.942 INFO:tasks.workunit.client.0.smithi116.stderr:192861: 3' 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:20:50.943 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 4 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: osd.2 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6810/778594144,v1:127.0.0.1:6811/778594144] [v2:127.0.0.1:6812/778594144,v1:127.0.0.1:6813/778594144] exists,up d299a592-234c-441b-9dec-c8415fa94bfa 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:20:50.944 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:20:50.945 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 1 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 2 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: 3' 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-06-15T10:20:50.946 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:53.158 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:20:53.158 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-15T10:20:53.158 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-15T10:20:53.158 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-317827579906' 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-317827579906 2-313532612610' 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-317827579906 2-313532612610 3-253403070476' 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:53.159 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: waiting osd.0 seq 21474836547 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836547 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:53.160 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836547 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579906 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579906 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /elpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-317827579907' 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-06-15T10:20:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-317827579907 2-313532612611' 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-317827579907 2-313532612611 3-253403070477' 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: waiting osd.0 seq 21474836546 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836546 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836546 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-06-15T10:20:53.163 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:53.164 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:20:53.164 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-06-15T10:20:53.164 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /homehome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: waiting osd.1 seq 317827579906 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579906 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:54.187 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: waiting osd.2 seq 313532612610 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612610 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:54.188 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: waiting osd.3 seq 253403070476 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070478 -lt 253403070476 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:54.189 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:54.190 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:54.193 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(cont/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-06-15T10:20:54.193 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-06-15T10:20:54.193 INFO:tasks.workunit.client.0.smithi116.stderr:192861: waiting osd.1 seq 317827579907 2024-06-15T10:20:54.193 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612611 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612611 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: waiting osd.2 seq 313532612611 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:20:54.194 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: waiting osd.3 seq 253403070477 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070478 -lt 253403070477 2024-06-15T10:20:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:20:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:20:54.647 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:192861: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:20:54.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 192862 2024-06-15T10:20:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:ains("stale") | not)' 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:192865: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:20:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:20:54.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:20:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:20:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:20:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:20:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:20:54.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:20:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 195588"' 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 195588' 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 195590"' 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 195590' 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 195588 195590' 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:20:54.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:20:54.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 195588 2024-06-15T10:20:54.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/195591: /' 2024-06-15T10:20:54.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:20:54.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:54.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/195593: /' 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:58.218 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:58.219 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:58.220 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:58.223 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:58.223 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:58.223 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: start osd.1 2024-06-15T10:20:58.225 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:20:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:20:59.157 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:20:59.158 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:20:59.160 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:20:59.160 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:20:59.160 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:20:59.161 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: start osd.3 2024-06-15T10:20:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2024-06-15T10:20:58.266+0000 7fa14bf337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2024-06-15T10:20:58.266+0000 7fa14bf337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2024-06-15T10:20:58.270+0000 7fa14bf337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:21:02.696 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 0 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2024-06-15T10:20:58.834+0000 7fa14bf337c0 -1 Falling back to public interface 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:02.697 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 1 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2024-06-15T10:21:02.210+0000 7fa14bf337c0 -1 osd.1 77 log_to_monitors true 2024-06-15T10:21:02.698 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:02.699 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:02.699 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:21:02.699 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 3 2024-06-15T10:21:02.699 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:02.699 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2024-06-15T10:20:59.194+0000 7f59e3fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2024-06-15T10:20:59.198+0000 7f59e3fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2024-06-15T10:20:59.198+0000 7f59e3fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:21:03.582 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 0 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2024-06-15T10:20:59.738+0000 7f59e3fdb7c0 -1 Falling back to public interface 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:21:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 1 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2024-06-15T10:21:02.806+0000 7f59e3fdb7c0 -1 osd.3 77 log_to_monitors true 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:21:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 3 2024-06-15T10:21:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:05.045 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:05.045 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 4 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: osd.1 up in weight 1 up_from 82 up_thru 74 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/2452473586,v1:127.0.0.1:6819/2452473586] [v2:127.0.0.1:6820/2452473586,v1:127.0.0.1:6821/2452473586] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:21:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:21:05.047 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 1 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 2 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: 3' 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-06-15T10:21:05.048 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:05.996 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hels.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 4 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: osd.3 up in weight 1 up_from 85 up_thru 74 down_at 78 last_clean_interval [59,77) [v2:127.0.0.1:6826/2173513691,v1:127.0.0.1:6827/2173513691] [v2:127.0.0.1:6828/2173513691,v1:127.0.0.1:6829/2173513691] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:21:05.997 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:21:05.998 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 1 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 2 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: 3' 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836552 2024-06-15T10:21:05.999 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836552 2024-06-15T10:21:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552' 2024-06-15T10:21:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:07.557 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:21:07.557 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-06-15T10:21:07.557 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-06-15T10:21:07.557 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-352187318275' 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612616 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612616 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-352187318275 2-313532612616' 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-352187318275 2-313532612616 3-365072220163' 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:07.558 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836552 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836552 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836552 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: waiting osd.0 seq 21474836552 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836552 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318275 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:07.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318275 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: waiting osd.1 seq 352187318275 2024-06-15T10:21:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:21:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221pers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:21:08.093 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-352187318274' 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612615 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612615 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-352187318274 2-313532612615' 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-06-15T10:21:08.094 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-352187318274 2-313532612615 3-365072220162' 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: waiting osd.0 seq 21474836551 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:21:08.095 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836551 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-06-15T10:21:08.096 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/u4: flush_pg_stats: test 352187318275 -lt 352187318275 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612616 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612616 2024-06-15T10:21:08.559 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612616 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612616' 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: waiting osd.2 seq 313532612616 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612616 -lt 313532612616 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-06-15T10:21:08.560 INFO:tasks.workunit.client.0.smithi116.stderr:195591: waiting osd.3 seq 365072220163 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:08.561 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_cleabuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195593: waiting osd.1 seq 352187318274 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318274 2024-06-15T10:21:09.229 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612615 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612615 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612615 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612615' 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: waiting osd.2 seq 313532612615 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612616 -lt 313532612615 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:09.230 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: waiting osd.3 seq 365072220162 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:09.231 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:21:09.232 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:09.232 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:09.232 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:09.232 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:09.421 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("n: cur_active_clean=4 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:09.422 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:21:09.423 INFO:tasks.workunit.client.0.smithi116.stderr:195591: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:21:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:21:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:21:09.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 195590 2024-06-15T10:21:09.742 INFO:tasks.workunit.client.0.smithi116.stderr:stale") | not)' 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:195593: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:21:09.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:21:09.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:21:09.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:21:09.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:21:09.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:21:09.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:21:09.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:21:09.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:21:09.745 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:21:10.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:21:10.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:21:10.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:21:10.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:21:10.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:10.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:10.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:10.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:10.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:10.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:21:10.491 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:21:10.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:10.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:21:10.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:10.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:10.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:10.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:34.727829+0000 '>' 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:10.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:21:11.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:21:11.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:21:11.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:21:11.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:11.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:11.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:11.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:11.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:34.727829+0000 '>' 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:11.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:21:12.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:21:12.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:21:12.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:21:12.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:12.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:12.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:12.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:13.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:34.727829+0000 '>' 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:13.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:21:14.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:21:14.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:21:14.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:21:14.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:14.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:14.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:14.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:14.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:20:34.727829+0000 '>' 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:14.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:21:15.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:21:15.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:21:15.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:21:15.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:21:15.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:21:15.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:21:15.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:21:10.899618+0000 '>' 2024-06-15T10:20:34.727829+0000 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:21:15.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 198515"' 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 198515' 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 198517"' 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 198517' 2024-06-15T10:21:15.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/198518: /' 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 198515 198517' 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 198515 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:21:15.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/198520: /' 2024-06-15T10:21:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:21:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:21:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:21:19.792 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:21:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: _ 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: hinfo_key 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: snapset 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:21:19.794 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:21:19.795 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:21:19.795 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:21:19.795 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:21:19.796 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:21:19.796 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:21:19.796 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:21:19.796 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:21:19.796 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:19.797 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:21:19.798 INFO:tasks.workunit.client.0.smithi116.stderr:198518: start osd.3 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:21:19.943 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:21:19.944 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: _ 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: hinfo_key 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: snapset 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:21:19.945 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:21:19.946 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:21:19.947 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:21:19.948 INFO:tasks.workunit.client.0.smithi116.stderr:198520: start osd.1 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:21:23.835 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2024-06-15T10:21:19.830+0000 7fda79eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2024-06-15T10:21:19.834+0000 7fda79eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2024-06-15T10:21:19.838+0000 7fda79eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 0 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:23.836 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2024-06-15T10:21:20.398+0000 7fda79eda7c0 -1 Falling back to public interface 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 1 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:23.837 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 3 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:23.838 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:24.424 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2024-06-15T10:21:23.834+0000 7fda7overy_ops 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2024-06-15T10:21:19.978+0000 7f504d7c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2024-06-15T10:21:19.978+0000 7f504d7c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2024-06-15T10:21:19.982+0000 7f504d7c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:21:24.425 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 0 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2024-06-15T10:21:20.534+0000 7f504d7c47c0 -1 Falling back to public interface 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:24.426 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 1 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:24.427 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2024-06-15T10:21:23.386+0000 7f504d7c47c0 -1 osd.1 87 log_to_monitors true 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 3 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:24.428 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:26.184 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c9eda7c0 -1 osd.3 87 log_to_monitors true 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 4 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: osd.3 up in weight 1 up_from 94 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6818/2994761630,v1:127.0.0.1:6819/2994761630] [v2:127.0.0.1:6820/2994761630,v1:127.0.0.1:6821/2994761630] exists,up c5998d60-97a0-410f-a3a9-3a3c8d2fa588 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:21:26.185 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:21:26.186 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 1 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 2 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: 3' 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:21:26.187 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:26.188 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:21:26.188 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-06-15T10:21:26.188 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-06-15T10:21:26.188 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-06-15T10:21:26.937 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperseph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:26.937 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:26.937 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 4 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: osd.1 up in weight 1 up_from 93 up_thru 93 down_at 88 last_clean_interval [82,87) [v2:127.0.0.1:6826/3541832513,v1:127.0.0.1:6827/3541832513] [v2:127.0.0.1:6828/3541832513,v1:127.0.0.1:6829/3541832513] exists,up f2751b12-3044-4cf9-ab75-7d3581db1099 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:21:26.938 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:21:26.939 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 1 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 2 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: 3' 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-06-15T10:21:26.940 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:27.587 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalon.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:27.587 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:21:27.587 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-399431958530' 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612621 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612621 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-399431958530 2-313532612621' 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-399431958530 2-313532612621 3-403726925826' 2024-06-15T10:21:27.588 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: waiting osd.0 seq 21474836557 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958530 2024-06-15T10:21:27.589 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958530 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-06-15T10:21:27.590 INFO:tasks.workunit.client.0.smithi116.stderr:198518: waiting osd.1 seq 399431958530 2024-06-15T10:21:29.274 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_e/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-399431958531' 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612622 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612622 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-399431958531 2-313532612622' 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:21:29.275 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-399431958531 2-313532612622 3-403726925827' 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:29.276 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: waiting osd.0 seq 21474836558 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836558 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836558 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958531 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:21:29.277 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958531 2024-06-15T10:21:29.278 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:19852stats: ceph osd last-stat-seq 1 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612621 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612621 2024-06-15T10:21:30.169 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612621 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612621' 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: waiting osd.2 seq 313532612621 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612620 -lt 313532612621 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612623 -lt 313532612621 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-403726925826 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:30.170 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-403726925826 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 403726925826' 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: waiting osd.3 seq 403726925826 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:30.171 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:21:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:30.172 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:30.304 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/0: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-06-15T10:21:30.304 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: waiting osd.1 seq 399431958531 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612622 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612622 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612622 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612622' 2024-06-15T10:21:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:198520: waiting osd.2 seq 313532612622 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612623 -lt 313532612622 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-403726925827 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-403726925827 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:21:30.306 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 403726925827' 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: waiting osd.3 seq 403726925827 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:30.310 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:30.311 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:31.375 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:21:31.376 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:31.377 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:21:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:31.379 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:31.379 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:31.379 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:31.490 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //homecontains("stale") | not)' 2024-06-15T10:21:31.490 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:31.490 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:31.490 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:31.491 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:31.492 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:31.493 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_cle/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:32.285 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:21:32.286 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:32.287 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:32.381 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/can 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:32.382 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:33.418 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:33.418 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:33.418 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:33.418 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:33.418 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-06-15T10:21:33.419 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:33.421 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:21:33.421 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:33.503 INFO:tasks.workunit.client.0.smithi116.stderr:1| .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:33.503 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:33.503 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:21:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=0 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 0 '!=' null 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:21:33.505 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:21:33.506 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:21:33.506 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --f98518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:198518: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:198518: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:21:33.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:21:33.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 198517 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:ormat json status 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:198520: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:198520: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:21:33.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:21:33.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:21:33.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:21:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:21:33.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:21:33.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:21:34.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:21:34.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:21:34.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:21:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:21:34.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:21:34.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:21:34.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:21:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:21:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:21:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:21:34.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:21:34.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:21:34.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:21:34.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:21:34.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:21:34.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:21:34.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:21:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:21:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:21:34.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:21:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:21:34.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:21:34.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:21:34.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:21:34.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:21:34.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:21:34.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:21:34.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:21:34.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:21:34.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:21:34.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:21:34.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:21:34.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:21:34.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:21:34.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:21:34.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:21:34.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:21:34.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:21:34.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:21:34.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:21:34.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:21:34.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:21:34.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:21:34.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:21:34.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:21:34.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:21:34.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:21:34.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:21:34.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.637 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:34.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:21:34.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:21:34.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:21:34.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:21:34.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:21:34.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:21:34.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:21:34.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:21:34.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:21:34.867 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:21:34.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:21:34.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:21:34.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:21:34.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:21:34.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:21:34.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:21:34.980 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:21:34.980 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:21:34.980 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:21:34.981 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:21:34.981 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:34.981 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:34.981 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:21:34.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:21:34.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:21:35.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-15T10:21:35.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:21:35.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:21:35.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:21:35.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:21:35.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:21:35.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:21:35.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:21:35.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:21:35.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:21:35.982 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:21:35.982 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:35.982 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:35.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:35.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:21:35.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:21:36.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-15T10:21:36.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:21:36.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:36.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:21:36.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:21:36.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:21:36.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=859d58f8-9a74-4a93-a324-457cc07a2f87 2024-06-15T10:21:36.029 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 859d58f8-9a74-4a93-a324-457cc07a2f87 2024-06-15T10:21:36.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 859d58f8-9a74-4a93-a324-457cc07a2f87' 2024-06-15T10:21:36.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:21:36.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwa21mU2W1AhAAvRkV1d1rH1mftpwLe+bljg== 2024-06-15T10:21:36.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwa21mU2W1AhAAvRkV1d1rH1mftpwLe+bljg=="}' 2024-06-15T10:21:36.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 859d58f8-9a74-4a93-a324-457cc07a2f87 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:21:36.342 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:21:36.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:21:36.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwa21mU2W1AhAAvRkV1d1rH1mftpwLe+bljg== --osd-uuid 859d58f8-9a74-4a93-a324-457cc07a2f87 2024-06-15T10:21:36.394 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.393+0000 7ff9d24307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:36.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.417+0000 7ff9d24307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:36.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.421+0000 7ff9d24307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:36.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.421+0000 7ff9d24307c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:21:36.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.421+0000 7ff9d24307c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:21:36.420 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.421+0000 7ff9d24307c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:21:36.518 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:36.517+0000 7ff9d24307c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:21:44.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:21:44.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:21:44.162 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:21:44.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:21:44.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:21:44.434 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:21:44.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:21:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:21:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:21:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:21:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:21:44.466 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:44.465+0000 7f9dc55a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:44.468 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:44.469+0000 7f9dc55a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:44.470 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:44.469+0000 7f9dc55a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:21:44.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:44.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:21:44.661 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:21:44.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:44.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:44.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:45.046 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:45.045+0000 7f9dc55a87c0 -1 Falling back to public interface 2024-06-15T10:21:45.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:45.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:45.916 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:21:45.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:21:45.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:45.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:46.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:47.160 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:21:47.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:21:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:47.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:47.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:47.931 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:21:47.933+0000 7f9dc55a87c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:21:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:48.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:48.405 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:21:48.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:21:48.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:48.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:48.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:49.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:49.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:49.777 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:21:49.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:21:49.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:49.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:50.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:51.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:51.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:51.079 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:21:51.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:21:51.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:51.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:51.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:52.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:52.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:52.321 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:21:52.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:21:52.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:52.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:52.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:53.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:53.568 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:21:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:21:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:53.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:54.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:54.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:54.951 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:21:54.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:21:54.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:54.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:55.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:56.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:56.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:56.194 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:21:56.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:21:56.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:56.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:56.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:57.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:57.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:57.437 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:21:57.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:21:57.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:57.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:57.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:21:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:21:58.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:21:58.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:21:58.741 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:21:58.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:21:58.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:21:59.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:00.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:00.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:00.072 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:22:00.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:22:00.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:00.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:00.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:01.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:01.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:01.363 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:22:01.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:22:01.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:01.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:01.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:02.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:02.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:02.752 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:22:02.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:22:02.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:02.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:03.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:04.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:04.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:04.199 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:22:04.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:22:04.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:04.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:04.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:05.528 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:22:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:22:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:05.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:05.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:06.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:06.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:06.920 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:22:06.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:22:06.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:06.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:07.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:08.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:08.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:08.224 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:22:08.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:22:08.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:08.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:08.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:09.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:09.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:09.837 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:22:09.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:22:09.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:09.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:10.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:11.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:11.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:11.075 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:22:11.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:22:11.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:11.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:11.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:12.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:12.648 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:22:12.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:22:12.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:12.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:13.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:14.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:14.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:14.349 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:22:14.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:22:14.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:14.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:15.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:16.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:16.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:16.022 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:22:16.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:22:16.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:16.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:16.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:17.648 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:22:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:22:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:17.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:18.982 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:22:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:22:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:22:19.228 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1342743077,v1:127.0.0.1:6803/1342743077] [v2:127.0.0.1:6804/1342743077,v1:127.0.0.1:6805/1342743077] exists,up 859d58f8-9a74-4a93-a324-457cc07a2f87 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:22:19.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:22:19.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:22:19.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:22:19.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:22:19.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd3def9c-c656-43fa-9397-b3db3b88a9da 2024-06-15T10:22:19.235 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 dd3def9c-c656-43fa-9397-b3db3b88a9da 2024-06-15T10:22:19.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dd3def9c-c656-43fa-9397-b3db3b88a9da' 2024-06-15T10:22:19.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:22:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBba21mQNGNDxAA2hEC2d8pMiQchTXQPPedjA== 2024-06-15T10:22:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBba21mQNGNDxAA2hEC2d8pMiQchTXQPPedjA=="}' 2024-06-15T10:22:19.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd3def9c-c656-43fa-9397-b3db3b88a9da -i td/osd-scrub-repair/1/new.json 2024-06-15T10:22:19.564 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:22:19.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:22:19.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBba21mQNGNDxAA2hEC2d8pMiQchTXQPPedjA== --osd-uuid dd3def9c-c656-43fa-9397-b3db3b88a9da 2024-06-15T10:22:19.620 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.621+0000 7f7740cd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:19.675 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.673+0000 7f7740cd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:19.676 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.677+0000 7f7740cd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.677+0000 7f7740cd97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:22:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.677+0000 7f7740cd97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:22:19.677 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.677+0000 7f7740cd97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:22:19.765 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:19.765+0000 7f7740cd97c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:22:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:22:28.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:22:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:22:28.095 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:22:28.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:22:28.370 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:22:28.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:22:28.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:22:28.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:22:28.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:22:28.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:22:28.402 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:28.401+0000 7f39f43e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:28.446 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:28.445+0000 7f39f43e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:28.447 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:28.449+0000 7f39f43e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:22:28.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:22:28.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:22:28.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:22:28.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:22:28.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:22:28.600 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:22:28.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:28.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:22:28.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:28.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:28.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:29.018 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:29.017+0000 7f39f43e67c0 -1 Falling back to public interface 2024-06-15T10:22:29.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:29.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:29.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:22:29.845 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:22:29.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:29.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:30.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:31.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:31.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:31.087 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:22:31.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:22:31.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:31.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:31.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:32.043 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:22:32.045+0000 7f39f43e67c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:22:32.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:32.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:32.433 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:22:32.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:22:32.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:32.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:33.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:34.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:22:34.044 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:22:34.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:34.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:34.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:35.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:35.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:35.319 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:22:35.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:22:35.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:35.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:35.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:36.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:36.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:36.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:22:36.648 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:22:36.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:36.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:36.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:37.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:37.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:37.933 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:22:37.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:22:37.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:37.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:38.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:39.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:22:39.286 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:22:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:39.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:40.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:40.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:40.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:22:40.690 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:22:40.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:40.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:40.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:41.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:41.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:41.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:22:41.971 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:22:41.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:41.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:42.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:43.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:43.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:43.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:22:43.292 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:22:43.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:43.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:43.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:44.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:44.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:44.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:22:44.597 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:22:44.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:44.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:44.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:45.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:45.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:45.843 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:22:45.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:22:45.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:45.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:46.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:47.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:47.181 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:22:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:22:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:47.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:47.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:48.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:48.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:48.635 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:22:48.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:22:48.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:48.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:48.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:49.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:49.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:49.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:22:49.912 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:22:49.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:49.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:50.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:51.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:51.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:51.538 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:22:51.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:22:51.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:51.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:51.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:52.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:52.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:52.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:22:52.956 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:22:52.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:52.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:53.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:54.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:54.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:54.686 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:22:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:22:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:54.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:55.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:56.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:56.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:56.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:22:56.314 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:22:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:56.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:56.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:57.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:57.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:57.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:22:57.570 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:22:57.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:57.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:58.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:22:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:22:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:22:59.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:22:59.093 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:22:59.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:22:59.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:22:59.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:00.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:00.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:00.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:23:00.647 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:23:00.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:00.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:23:00.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:01.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:01.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:01.892 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:23:01.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:23:01.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:01.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:23:02.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:03.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:03.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:03.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:23:03.297 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:23:03.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:03.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:23:03.587 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3910060653,v1:127.0.0.1:6811/3910060653] [v2:127.0.0.1:6812/3910060653,v1:127.0.0.1:6813/3910060653] exists,up dd3def9c-c656-43fa-9397-b3db3b88a9da 2024-06-15T10:23:03.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:23:03.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:23:03.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:23:03.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:23:03.592 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:23:03.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:23:03.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:23:03.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:23:03.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:23:03.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=452efef0-898c-41f3-bd96-ed30911b8296 2024-06-15T10:23:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 452efef0-898c-41f3-bd96-ed30911b8296' 2024-06-15T10:23:03.595 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 452efef0-898c-41f3-bd96-ed30911b8296 2024-06-15T10:23:03.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:23:03.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHa21m6ajgJBAA+K8lVy8O7ydAZ3zWopWORA== 2024-06-15T10:23:03.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHa21m6ajgJBAA+K8lVy8O7ydAZ3zWopWORA=="}' 2024-06-15T10:23:03.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 452efef0-898c-41f3-bd96-ed30911b8296 -i td/osd-scrub-repair/2/new.json 2024-06-15T10:23:03.971 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:23:03.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:23:03.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHa21m6ajgJBAA+K8lVy8O7ydAZ3zWopWORA== --osd-uuid 452efef0-898c-41f3-bd96-ed30911b8296 2024-06-15T10:23:04.023 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.024+0000 7f094b8847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:04.106 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.104+0000 7f094b8847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:04.107 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.108+0000 7f094b8847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:04.108 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.108+0000 7f094b8847c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:23:04.108 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.108+0000 7f094b8847c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:23:04.108 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.108+0000 7f094b8847c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:23:04.191 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:04.192+0000 7f094b8847c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:23:12.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:23:12.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:23:12.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:23:12.223 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:23:12.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:23:12.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:23:12.499 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:23:12.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:23:12.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:23:12.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:23:12.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:23:12.531 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:12.532+0000 7f0f2b1757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:12.535 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:12.536+0000 7f0f2b1757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:12.537 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:12.536+0000 7f0f2b1757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:12.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:23:12.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:23:12.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:23:12.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:23:12.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:23:12.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:12.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:23:12.743 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:23:12.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:12.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:12.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:13.098 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:13.096+0000 7f0f2b1757c0 -1 Falling back to public interface 2024-06-15T10:23:13.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:13.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:13.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:23:13.972 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:23:13.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:13.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:14.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:15.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:15.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:15.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:23:15.214 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:23:15.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:15.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:15.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:16.027 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:16.028+0000 7f0f2b1757c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:23:16.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:23:16.455 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:23:16.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:16.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:17.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:23:18.078 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:23:18.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:18.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:18.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:19.511 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:23:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:23:19.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:19.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:19.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:20.844 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:23:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:23:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:21.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:22.197 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:23:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:23:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:22.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:22.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:23.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:23.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:23.438 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:23:23.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:23:23.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:23.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:23.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:24.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:24.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:24.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:23:24.680 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:23:24.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:24.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:24.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:25.979 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:23:25.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:23:25.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:25.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:26.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:27.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:27.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:27.335 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:23:27.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:23:27.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:27.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:27.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:28.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:28.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:28.736 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:23:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:23:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:28.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:29.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:30.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:30.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:30.027 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:23:30.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:23:30.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:30.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:30.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:31.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:31.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:31.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:23:31.474 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:23:31.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:31.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:31.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:32.899 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:23:32.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:23:32.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:32.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:33.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:34.332 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:23:34.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:23:34.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:34.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:34.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:35.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:35.960 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:23:35.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:23:35.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:35.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:36.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:37.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:37.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:37.227 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:23:37.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:23:37.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:37.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:37.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:38.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:38.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:38.870 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:23:38.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:23:38.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:38.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:39.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:40.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:40.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:40.507 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:23:40.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:23:40.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:40.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:41.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:42.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:42.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:42.173 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:23:42.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:23:42.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:42.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:42.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:43.887 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:23:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:23:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:44.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:45.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:45.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:45.237 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:23:45.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:23:45.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:45.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:45.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:46.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:46.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:46.924 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:23:46.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:23:46.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:46.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:47.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:48.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:48.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:48.192 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:23:48.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:23:48.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:48.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:23:48.521 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1618044987,v1:127.0.0.1:6819/1618044987] [v2:127.0.0.1:6820/1618044987,v1:127.0.0.1:6821/1618044987] exists,up 452efef0-898c-41f3-bd96-ed30911b8296 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:23:48.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:23:48.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:23:48.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:23:48.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:23:48.529 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:23:48.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 25307158-8c45-4982-8fa8-1f0fe831e9d5' 2024-06-15T10:23:48.529 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:23:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0a21mfiu1IBAAChaqj1xs6jU6CMqM9vuIvA== 2024-06-15T10:23:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0a21mfiu1IBAAChaqj1xs6jU6CMqM9vuIvA=="}' 2024-06-15T10:23:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25307158-8c45-4982-8fa8-1f0fe831e9d5 -i td/osd-scrub-repair/3/new.json 2024-06-15T10:23:48.846 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:23:48.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-15T10:23:48.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0a21mfiu1IBAAChaqj1xs6jU6CMqM9vuIvA== --osd-uuid 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:23:48.901 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.899+0000 7f62ef68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:48.973 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.971+0000 7f62ef68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.971+0000 7f62ef68b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.975+0000 7f62ef68b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:23:48.976 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.975+0000 7f62ef68b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:23:48.976 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:48.975+0000 7f62ef68b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:23:49.063 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:49.063+0000 7f62ef68b7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-15T10:23:56.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-15T10:23:56.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:23:56.958 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T10:23:56.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T10:23:56.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:23:57.231 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T10:23:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T10:23:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:23:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:23:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:23:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:23:57.263 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:57.263+0000 7f298bf4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:57.265 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:57.263+0000 7f298bf4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:57.267 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:57.267+0000 7f298bf4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:57.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:23:57.528 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:23:57.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:57.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:23:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:23:57.816 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:23:57.815+0000 7f298bf4d7c0 -1 Falling back to public interface 2024-06-15T10:23:58.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:23:58.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:23:58.761 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:23:58.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:23:58.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:23:58.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:23:58.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:00.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:00.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:00.001 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:24:00.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:24:00.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:00.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:00.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:00.752 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:00.751+0000 7f298bf4d7c0 -1 osd.3 0 log_to_monitors true 2024-06-15T10:24:01.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:01.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:01.242 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:24:01.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:24:01.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:01.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:01.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:02.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:02.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:02.703 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:24:02.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:24:02.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:02.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:03.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:04.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:04.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:04.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:24:04.102 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:24:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:04.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:05.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:05.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:05.348 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:24:05.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:24:05.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:05.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:05.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:06.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:06.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:06.714 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:24:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:24:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:07.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:08.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:08.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:08.099 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:24:08.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:24:08.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:08.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:08.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:09.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:09.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:09.343 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:24:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:24:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:09.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:09.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:10.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:10.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:10.786 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:24:10.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:24:10.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:10.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:11.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:12.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:12.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:12.027 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:24:12.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:24:12.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:12.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:12.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:13.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:13.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:13.268 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:24:13.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:24:13.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:13.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:13.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:14.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:14.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:14.712 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:24:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:24:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:14.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:15.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:16.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:16.059 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:24:16.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:24:16.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:16.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:16.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:17.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:17.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:17.299 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:24:17.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:24:17.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:17.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:17.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:18.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:18.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:18.670 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:24:18.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:24:18.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:18.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:18.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:19.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:19.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:19.993 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:24:19.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:24:19.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:19.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:20.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:21.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:21.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:21.239 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:24:21.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:24:21.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:21.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:21.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:22.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:22.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:22.869 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:24:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:24:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:22.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:23.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:24.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:24.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:24.543 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:24:24.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:24:24.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:24.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:24.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:25.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:25.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:25.830 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:24:25.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:24:25.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:25.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:26.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:27.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:27.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:27.530 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:24:27.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:24:27.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:27.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:27.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:28.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:28.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:28.773 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:24:28.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:24:28.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:28.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:29.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:30.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:30.336 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:24:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:24:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:30.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:31.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:32.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:32.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:32.073 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:24:32.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:24:32.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:32.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:24:32.314 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3444554384,v1:127.0.0.1:6827/3444554384] [v2:127.0.0.1:6828/3444554384,v1:127.0.0.1:6829/3444554384] exists,up 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-15T10:24:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:24:32.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:24:32.320 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:24:32.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:24:32.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a60b26a6-c478-4842-bcbe-5947b90d24c3 2024-06-15T10:24:32.323 INFO:tasks.workunit.client.0.smithi116.stdout:add osd4 a60b26a6-c478-4842-bcbe-5947b90d24c3 2024-06-15T10:24:32.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a60b26a6-c478-4842-bcbe-5947b90d24c3' 2024-06-15T10:24:32.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:24:32.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDga21mYiJ6FBAAPbnPVqhwanT1p/XLl1nF1Q== 2024-06-15T10:24:32.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDga21mYiJ6FBAAPbnPVqhwanT1p/XLl1nF1Q=="}' 2024-06-15T10:24:32.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a60b26a6-c478-4842-bcbe-5947b90d24c3 -i td/osd-scrub-repair/4/new.json 2024-06-15T10:24:32.695 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:24:32.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-15T10:24:32.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDga21mYiJ6FBAAPbnPVqhwanT1p/XLl1nF1Q== --osd-uuid a60b26a6-c478-4842-bcbe-5947b90d24c3 2024-06-15T10:24:32.753 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.755+0000 7f34e410d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:32.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.823+0000 7f34e410d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:32.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.827+0000 7f34e410d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:32.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.827+0000 7f34e410d7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:24:32.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.827+0000 7f34e410d7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:24:32.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.827+0000 7f34e410d7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:24:32.911 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:32.915+0000 7f34e410d7c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-15T10:24:41.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-15T10:24:41.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:24:41.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-15T10:24:41.087 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd4 key to auth repository 2024-06-15T10:24:41.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:24:41.377 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.4 2024-06-15T10:24:41.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-15T10:24:41.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:24:41.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:24:41.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:24:41.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:24:41.410 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:41.410+0000 7f5e4e8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:41.464 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:41.466+0000 7f5e4e8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:41.466 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:41.466+0000 7f5e4e8f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:24:41.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:24:41.610 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:24:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:41.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:41.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:42.022 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:42.022+0000 7f5e4e8f57c0 -1 Falling back to public interface 2024-06-15T10:24:42.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:42.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:42.855 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:24:42.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:24:42.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:42.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:43.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:44.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:44.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:44.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:24:44.095 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:24:44.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:44.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:44.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:45.003 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:24:45.006+0000 7f5e4e8f57c0 -1 osd.4 0 log_to_monitors true 2024-06-15T10:24:45.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:45.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:45.334 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:24:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:24:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:45.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:45.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:46.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:46.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:46.710 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:24:46.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:24:46.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:46.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:47.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:48.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:48.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:48.134 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:24:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:24:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:48.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:48.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:49.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:49.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:49.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:24:49.421 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:24:49.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:49.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:49.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:50.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:50.759 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:24:50.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:24:50.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:50.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:51.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:52.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:52.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:52.147 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:24:52.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:24:52.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:52.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:52.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:53.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:53.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:53.508 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:24:53.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:24:53.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:53.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:53.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:54.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:54.863 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:24:54.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:24:54.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:54.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:55.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:24:56.297 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:24:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:56.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:57.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:57.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:57.585 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:24:57.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:24:57.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:57.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:57.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:24:58.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:24:58.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:24:58.895 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:24:58.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:24:58.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:24:58.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:24:59.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:00.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:00.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:00.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:25:00.280 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:25:00.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:00.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:00.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:01.591 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:25:01.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:25:01.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:01.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:01.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:02.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:02.851 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:25:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:25:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:02.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:03.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:04.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:04.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:04.237 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:25:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:25:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:04.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:05.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:05.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:05.954 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:25:05.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:25:05.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:05.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:06.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:07.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:07.271 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:25:07.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:25:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:07.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:07.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:08.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:25:08.954 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:25:08.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:08.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:09.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:10.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:10.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:10.255 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:25:10.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:25:10.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:10.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:10.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:11.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:11.797 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:25:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:25:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:11.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:12.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:13.253 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:25:13.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:25:13.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:13.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:13.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:14.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:14.652 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:25:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:25:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:14.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:15.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:15.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:15.896 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:25:15.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:25:15.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:15.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:16.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:17.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:17.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:17.388 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:25:17.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:25:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:17.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:25:17.643 INFO:tasks.workunit.client.0.smithi116.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/28093180,v1:127.0.0.1:6835/28093180] [v2:127.0.0.1:6836/28093180,v1:127.0.0.1:6837/28093180] exists,up a60b26a6-c478-4842-bcbe-5947b90d24c3 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:25:17.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:25:17.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:25:17.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:25:17.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:25:17.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:25:17.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:25:17.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:25:17.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f' 2024-06-15T10:25:17.650 INFO:tasks.workunit.client.0.smithi116.stdout:add osd5 b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:25:17.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:25:17.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANbG1muqdAKBAA/pfJz/bE4lLxV/z3/Qaatw== 2024-06-15T10:25:17.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANbG1muqdAKBAA/pfJz/bE4lLxV/z3/Qaatw=="}' 2024-06-15T10:25:17.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f -i td/osd-scrub-repair/5/new.json 2024-06-15T10:25:18.031 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:25:18.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-15T10:25:18.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANbG1muqdAKBAA/pfJz/bE4lLxV/z3/Qaatw== --osd-uuid b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:25:18.087 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.086+0000 7f5e684807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:18.174 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.174+0000 7f5e684807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:18.176 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.178+0000 7f5e684807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:18.177 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.178+0000 7f5e684807c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:25:18.177 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.178+0000 7f5e684807c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:25:18.177 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.178+0000 7f5e684807c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:25:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:18.282+0000 7f5e684807c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-15T10:25:26.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-15T10:25:26.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:25:26.746 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd5 key to auth repository 2024-06-15T10:25:26.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-15T10:25:26.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:25:27.014 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.5 2024-06-15T10:25:27.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-15T10:25:27.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:25:27.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:25:27.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:25:27.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:25:27.045 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:27.046+0000 7ff1572547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:27.047 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:27.050+0000 7ff1572547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:27.049 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:27.050+0000 7ff1572547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:25:27.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:25:27.247 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:25:27.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:27.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:27.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:27.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:27.586+0000 7ff1572547c0 -1 Falling back to public interface 2024-06-15T10:25:28.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:28.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:28.491 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:25:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:25:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:28.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:28.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:29.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:29.733 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:25:29.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:25:29.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:29.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:29.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:30.583 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:25:30.586+0000 7ff1572547c0 -1 osd.5 0 log_to_monitors true 2024-06-15T10:25:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:30.977 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:25:30.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:25:30.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:30.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:31.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:32.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:32.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:32.557 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:25:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:25:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:32.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:32.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:33.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:33.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:33.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:25:33.984 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:25:33.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:33.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:34.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:35.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:35.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:35.226 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:25:35.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:25:35.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:35.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:35.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:36.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:36.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:36.599 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:25:36.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:25:36.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:36.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:36.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:37.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:37.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:37.945 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:25:37.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:25:37.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:37.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:38.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:39.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:39.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:39.187 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:25:39.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:25:39.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:39.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:39.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:40.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:40.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:40.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:25:40.591 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:25:40.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:40.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:41.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:41.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:41.927 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:25:41.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:25:41.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:41.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:42.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:43.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:43.168 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:25:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:25:43.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:43.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:43.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:44.507 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:25:44.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:25:44.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:44.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:44.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:45.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:45.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:45.923 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:25:45.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:25:45.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:45.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:46.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:47.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:47.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:47.163 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:25:47.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:25:47.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:47.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:47.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:48.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:48.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:48.576 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:25:48.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:25:48.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:48.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:49.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:50.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:50.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:50.017 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:25:50.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:25:50.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:50.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:50.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:51.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:51.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:51.259 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:25:51.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:25:51.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:51.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:52.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:53.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:53.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:53.006 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:25:53.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:25:53.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:53.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:53.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:54.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:54.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:54.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:25:54.643 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:25:54.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:54.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:54.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:55.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:55.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:55.886 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:25:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:25:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:57.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:57.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:57.666 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:25:57.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:25:57.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:57.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:58.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:25:59.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:25:59.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:25:59.214 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:25:59.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:25:59.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:25:59.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:25:59.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:00.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:00.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:00.940 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:26:00.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:26:00.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:00.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:26:01.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:02.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:02.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:02.564 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:26:02.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:26:02.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:02.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:26:02.809 INFO:tasks.workunit.client.0.smithi116.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3827888193,v1:127.0.0.1:6843/3827888193] [v2:127.0.0.1:6844/3827888193,v1:127.0.0.1:6845/3827888193] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:26:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:26:02.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:26:02.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:26:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=671c2538-ed60-4ebd-a5f0-065c5e0700b3 2024-06-15T10:26:02.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 671c2538-ed60-4ebd-a5f0-065c5e0700b3' 2024-06-15T10:26:02.816 INFO:tasks.workunit.client.0.smithi116.stdout:add osd6 671c2538-ed60-4ebd-a5f0-065c5e0700b3 2024-06-15T10:26:02.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:26:02.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6bG1mihTGMRAAjnkghGsSLfsribYyEVWAIQ== 2024-06-15T10:26:02.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6bG1mihTGMRAAjnkghGsSLfsribYyEVWAIQ=="}' 2024-06-15T10:26:02.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 671c2538-ed60-4ebd-a5f0-065c5e0700b3 -i td/osd-scrub-repair/6/new.json 2024-06-15T10:26:03.163 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:26:03.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-15T10:26:03.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6bG1mihTGMRAAjnkghGsSLfsribYyEVWAIQ== --osd-uuid 671c2538-ed60-4ebd-a5f0-065c5e0700b3 2024-06-15T10:26:03.219 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.221+0000 7fc49a7307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:03.239 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.241+0000 7fc49a7307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:03.241 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.241+0000 7fc49a7307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:03.241 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.241+0000 7fc49a7307c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:26:03.242 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.241+0000 7fc49a7307c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:26:03.242 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.241+0000 7fc49a7307c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:26:03.390 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:03.389+0000 7fc49a7307c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-15T10:26:10.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-15T10:26:10.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:26:10.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-15T10:26:10.435 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd6 key to auth repository 2024-06-15T10:26:10.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:26:10.707 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.6 2024-06-15T10:26:10.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-15T10:26:10.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:26:10.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:26:10.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:26:10.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:26:10.738 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:10.737+0000 7f03da78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:10.741 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:10.741+0000 7f03da78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:10.742 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:10.741+0000 7f03da78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:10.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:26:10.936 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:26:10.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:10.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:11.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:11.277 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:11.277+0000 7f03da78b7c0 -1 Falling back to public interface 2024-06-15T10:26:12.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:12.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:12.174 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:26:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:26:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:12.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:13.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:13.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:13.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:26:13.428 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:26:13.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:13.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:13.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:13.932 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:13.933+0000 7f03da78b7c0 -1 osd.6 0 log_to_monitors true 2024-06-15T10:26:14.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:14.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:14.670 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:26:14.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:26:14.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:14.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:16.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:16.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:16.183 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:26:16.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:26:16.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:16.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:16.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:17.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:17.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:17.468 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:26:17.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:26:17.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:17.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:17.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:18.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:18.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:18.891 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:26:18.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:26:18.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:18.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:20.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:20.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:20.217 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:26:20.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:26:20.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:20.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:20.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:21.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:21.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:21.486 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:26:21.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:26:21.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:21.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:21.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:22.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:22.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:22.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:26:22.875 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:26:22.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:22.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:23.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:24.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:24.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:24.184 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:26:24.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:26:24.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:24.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:24.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:25.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:25.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:25.496 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:26:25.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:26:25.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:25.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:25.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:26.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:26.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:26.741 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:26:26.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:26:26.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:26.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:26.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:27.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:27.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:27.980 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:26:27.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:26:27.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:27.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:28.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:29.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:29.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:29.222 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:26:29.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:26:29.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:29.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:29.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:30.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:30.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:30.519 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:26:30.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:26:30.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:30.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:30.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:31.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:31.763 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:26:31.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:26:31.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:31.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:32.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:33.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:33.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:33.009 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:26:33.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:26:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:33.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:34.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:34.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:34.447 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:26:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:26:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:34.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:35.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:35.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:35.948 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:26:35.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:26:35.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:35.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:36.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:37.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:37.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:37.510 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:26:37.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:26:37.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:37.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:37.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:38.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:38.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:38.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:26:38.929 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:26:38.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:38.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:39.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:40.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:40.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:40.261 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:26:40.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:26:40.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:40.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:40.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:41.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:41.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:41.901 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:26:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:26:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:42.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:43.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:43.242 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:26:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:26:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:43.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:43.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:44.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:44.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:44.687 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:26:44.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:26:44.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:44.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:26:44.929 INFO:tasks.workunit.client.0.smithi116.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4177775455,v1:127.0.0.1:6851/4177775455] [v2:127.0.0.1:6852/4177775455,v1:127.0.0.1:6853/4177775455] exists,up 671c2538-ed60-4ebd-a5f0-065c5e0700b3 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:26:44.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:26:44.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:26:44.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:26:44.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39fd10db-2050-4ed1-ada0-868d3c4a2888 2024-06-15T10:26:44.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 39fd10db-2050-4ed1-ada0-868d3c4a2888' 2024-06-15T10:26:44.942 INFO:tasks.workunit.client.0.smithi116.stdout:add osd7 39fd10db-2050-4ed1-ada0-868d3c4a2888 2024-06-15T10:26:44.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:26:44.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkbG1mh1Y+ORAAEGvlDUUMlUDZpvo90QsQQA== 2024-06-15T10:26:44.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkbG1mh1Y+ORAAEGvlDUUMlUDZpvo90QsQQA=="}' 2024-06-15T10:26:44.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39fd10db-2050-4ed1-ada0-868d3c4a2888 -i td/osd-scrub-repair/7/new.json 2024-06-15T10:26:45.634 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:26:45.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-15T10:26:45.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkbG1mh1Y+ORAAEGvlDUUMlUDZpvo90QsQQA== --osd-uuid 39fd10db-2050-4ed1-ada0-868d3c4a2888 2024-06-15T10:26:45.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.688+0000 7fdf13b747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:45.797 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.796+0000 7fdf13b747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:45.799 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.800+0000 7fdf13b747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:45.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.800+0000 7fdf13b747c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:26:45.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.800+0000 7fdf13b747c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:26:45.800 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.800+0000 7fdf13b747c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:26:45.851 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:45.848+0000 7fdf13b747c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-15T10:26:53.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-15T10:26:53.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:26:53.424 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd7 key to auth repository 2024-06-15T10:26:53.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-15T10:26:53.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:26:53.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-15T10:26:53.699 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.7 2024-06-15T10:26:53.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:26:53.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:26:53.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:26:53.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:26:53.730 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:53.728+0000 7f35a12b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:53.779 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:53.780+0000 7f35a12b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:53.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:53.780+0000 7f35a12b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:26:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-15T10:26:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:26:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-15T10:26:53.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:26:53.925 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:26:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:26:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:26:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:53.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:26:54.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:54.342 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:54.340+0000 7f35a12b27c0 -1 Falling back to public interface 2024-06-15T10:26:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:55.172 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:26:55.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:26:55.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:55.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:26:55.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:56.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:56.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:56.414 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:26:56.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:26:56.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:56.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:26:56.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:57.192 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:26:57.192+0000 7f35a12b27c0 -1 osd.7 0 log_to_monitors true 2024-06-15T10:26:57.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:57.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:57.657 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:26:57.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:26:57.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:57.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:26:58.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:26:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:26:59.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:26:59.374 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:26:59.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:26:59.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:26:59.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:26:59.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:00.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:00.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:00.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:27:00.694 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:27:00.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:00.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:00.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:01.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:01.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:01.934 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:27:01.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:27:01.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:01.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:02.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:03.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:03.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:03.347 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:27:03.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:27:03.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:03.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:03.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:04.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:04.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:04.591 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:27:04.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:27:04.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:04.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:04.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:05.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:05.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:05.833 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:27:05.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:27:05.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:05.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:06.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:07.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:07.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:07.098 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:27:07.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:27:07.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:07.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:07.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:08.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:08.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:08.481 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:27:08.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:27:08.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:08.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:08.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:09.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:09.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:09.747 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:27:09.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:27:09.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:09.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:09.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:10.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:10.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:10.992 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:27:10.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:27:10.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:10.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:11.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:12.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:12.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:12.369 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:27:12.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:27:12.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:12.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:12.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:27:13.609 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:27:13.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:13.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:14.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:15.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:15.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:15.063 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:27:15.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:27:15.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:15.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:15.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:16.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:16.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:16.697 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:27:16.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:27:16.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:16.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:17.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:18.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:18.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:18.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:27:18.531 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:27:18.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:18.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:18.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:19.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:19.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:19.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:27:19.842 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:27:19.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:19.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:20.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:21.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:21.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:21.453 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:27:21.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:27:21.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:21.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:21.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:22.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:22.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:27:22.823 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:27:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:22.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:23.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:24.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:24.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:24.318 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:27:24.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:27:24.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:24.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:24.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:25.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:25.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:25.842 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:27:25.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:27:25.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:25.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:27:26.085 INFO:tasks.workunit.client.0.smithi116.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4247069256,v1:127.0.0.1:6859/4247069256] [v2:127.0.0.1:6860/4247069256,v1:127.0.0.1:6861/4247069256] exists,up 39fd10db-2050-4ed1-ada0-868d3c4a2888 2024-06-15T10:27:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:27:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:27:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:27:26.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-15T10:27:26.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:27:26.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:27:26.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-15T10:27:26.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:27:26.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30c26a4b-21f9-4d4f-b120-6635b645882e 2024-06-15T10:27:26.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 30c26a4b-21f9-4d4f-b120-6635b645882e' 2024-06-15T10:27:26.092 INFO:tasks.workunit.client.0.smithi116.stdout:add osd8 30c26a4b-21f9-4d4f-b120-6635b645882e 2024-06-15T10:27:26.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:27:26.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCObG1mGIb3BhAA9QBuCcF87EvPnF6Vk5PzfQ== 2024-06-15T10:27:26.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCObG1mGIb3BhAA9QBuCcF87EvPnF6Vk5PzfQ=="}' 2024-06-15T10:27:26.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30c26a4b-21f9-4d4f-b120-6635b645882e -i td/osd-scrub-repair/8/new.json 2024-06-15T10:27:26.683 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:27:26.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-15T10:27:26.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCObG1mGIb3BhAA9QBuCcF87EvPnF6Vk5PzfQ== --osd-uuid 30c26a4b-21f9-4d4f-b120-6635b645882e 2024-06-15T10:27:26.743 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.740+0000 7f4e19b1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:26.886 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.884+0000 7f4e19b1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:26.888 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.888+0000 7f4e19b1e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.888+0000 7f4e19b1e7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:27:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.888+0000 7f4e19b1e7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:27:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:26.888+0000 7f4e19b1e7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:27:27.018 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:27.016+0000 7f4e19b1e7c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-15T10:27:35.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-15T10:27:35.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:27:35.486 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd8 key to auth repository 2024-06-15T10:27:35.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-15T10:27:35.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:27:35.760 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.8 2024-06-15T10:27:35.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-15T10:27:35.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:27:35.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:27:35.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:27:35.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:27:35.787 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:35.788+0000 7fe1f4ba67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:35.789 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:35.788+0000 7fe1f4ba67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:35.791 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:35.792+0000 7fe1f4ba67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:27:36.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-15T10:27:36.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:27:36.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-15T10:27:36.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:27:36.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:27:36.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:36.044 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:27:36.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:27:36.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:36.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:36.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:36.357 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:36.356+0000 7fe1f4ba67c0 -1 Falling back to public interface 2024-06-15T10:27:37.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:37.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:37.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:27:37.259 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:27:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:37.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:37.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:38.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:38.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:38.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:27:38.500 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:27:38.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:38.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:38.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:39.477 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:27:39.476+0000 7fe1f4ba67c0 -1 osd.8 0 log_to_monitors true 2024-06-15T10:27:39.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:39.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:39.740 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:27:39.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:27:39.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:39.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:40.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:41.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:41.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:41.134 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:27:41.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:27:41.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:41.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:41.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:42.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:42.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:42.572 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:27:42.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:27:42.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:42.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:42.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:43.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:43.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:43.838 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:27:43.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:27:43.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:43.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:44.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:45.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:45.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:45.079 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:27:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:27:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:45.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:45.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:46.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:27:46.564 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:27:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:46.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:46.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:27:47.807 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:27:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:48.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:49.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:49.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:49.046 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:27:49.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:27:49.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:49.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:49.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:50.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:50.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:50.529 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:27:50.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:27:50.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:50.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:50.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:51.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:51.826 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:27:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:27:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:52.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:53.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:53.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:53.071 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:27:53.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:27:53.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:53.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:53.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:54.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:54.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:54.402 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:27:54.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:27:54.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:54.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:54.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:55.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:55.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:55.747 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:27:55.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:27:55.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:55.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:55.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:56.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:56.988 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:27:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:27:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:56.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:27:58.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:27:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:27:58.235 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:27:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:27:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:27:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:27:59.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:00.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:00.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:00.112 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:28:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:28:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:00.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:01.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:01.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:01.582 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:28:01.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:28:01.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:01.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:01.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:02.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:28:02.880 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:28:02.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:02.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:03.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:04.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:04.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:04.484 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:28:04.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:28:04.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:04.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:04.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:05.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:05.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:05.728 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:28:05.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:28:05.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:05.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:06.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:07.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:07.324 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:28:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:28:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:08.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:09.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:09.049 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:28:09.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:28:09.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:09.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:09.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:10.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:10.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:10.722 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:28:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:28:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:10.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:10.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:11.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:11.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:11.981 INFO:tasks.workunit.client.0.smithi116.stdout:26 2024-06-15T10:28:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-06-15T10:28:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:28:12.222 INFO:tasks.workunit.client.0.smithi116.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2006753115,v1:127.0.0.1:6867/2006753115] [v2:127.0.0.1:6868/2006753115,v1:127.0.0.1:6869/2006753115] exists,up 30c26a4b-21f9-4d4f-b120-6635b645882e 2024-06-15T10:28:12.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:28:12.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:28:12.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:28:12.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:28:12.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:28:12.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:28:12.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7' 2024-06-15T10:28:12.230 INFO:tasks.workunit.client.0.smithi116.stdout:add osd9 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:28:12.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:28:12.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8bG1m533ODhAAJeWAdfvGq3LRl3h6xwg12A== 2024-06-15T10:28:12.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8bG1m533ODhAAJeWAdfvGq3LRl3h6xwg12A=="}' 2024-06-15T10:28:12.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 -i td/osd-scrub-repair/9/new.json 2024-06-15T10:28:12.729 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:28:12.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-15T10:28:12.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8bG1m533ODhAAJeWAdfvGq3LRl3h6xwg12A== --osd-uuid 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:28:12.788 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.787+0000 7fc10dd427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:12.911 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.911+0000 7fc10dd427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:12.913 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.911+0000 7fc10dd427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.911+0000 7fc10dd427c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:28:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.911+0000 7fc10dd427c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:28:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:12.911+0000 7fc10dd427c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:28:13.137 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:13.135+0000 7fc10dd427c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-15T10:28:20.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-15T10:28:20.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:28:20.869 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd9 key to auth repository 2024-06-15T10:28:20.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-15T10:28:20.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:28:21.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-15T10:28:21.145 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.9 2024-06-15T10:28:21.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:28:21.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:28:21.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:28:21.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:28:21.177 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:21.175+0000 7f99f3e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:21.189 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:21.183+0000 7f99f3e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:21.190 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:21.187+0000 7f99f3e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:28:21.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:21.396 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:28:21.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:28:21.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:21.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:21.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:21.779+0000 7f99f3e947c0 -1 Falling back to public interface 2024-06-15T10:28:22.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:22.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:22.619 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:28:22.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:28:22.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:22.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:22.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:23.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:28:23.856 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:28:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:24.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:24.654 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:28:24.651+0000 7f99f3e947c0 -1 osd.9 0 log_to_monitors true 2024-06-15T10:28:25.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:25.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:25.090 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:28:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:28:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:25.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:25.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:26.555 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:28:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:28:26.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:26.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:27.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:28.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:28.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:28.061 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:28:28.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:28:28.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:28.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:28.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:29.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:29.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:29.295 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:28:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:28:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:29.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:30.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:30.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:30.649 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:28:30.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:28:30.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:30.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:31.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:32.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:32.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:32.335 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:28:32.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:28:32.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:32.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:33.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:33.606 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:28:33.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:28:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:33.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:33.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:34.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:34.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:34.848 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:28:34.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:28:34.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:34.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:35.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:36.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:36.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:36.091 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:28:36.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:28:36.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:36.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:36.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:37.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:37.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:37.334 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:28:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:28:37.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:37.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:37.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:38.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:38.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:38.645 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:28:38.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:28:38.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:38.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:38.952 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:39.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:39.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:39.955 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:28:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:28:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:40.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:41.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:41.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:41.199 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:28:41.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:28:41.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:41.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:41.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:42.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:42.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:42.574 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:28:42.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:28:42.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:42.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:44.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:44.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:44.303 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:28:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:28:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:44.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:45.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:45.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:45.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:28:45.835 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:28:45.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:45.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:46.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:47.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:47.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:47.221 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:28:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:28:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:47.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:48.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:48.511 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:28:48.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:28:48.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:48.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:48.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:49.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:49.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:49.987 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:28:49.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:28:49.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:49.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:50.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:51.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:51.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:51.410 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:28:51.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:28:51.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:51.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:51.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:52.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:52.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:52.723 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:28:52.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:28:52.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:52.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:53.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:54.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:54.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:54.410 INFO:tasks.workunit.client.0.smithi116.stdout:24 2024-06-15T10:28:54.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-06-15T10:28:54.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:54.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:54.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:28:55.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:28:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:28:55.898 INFO:tasks.workunit.client.0.smithi116.stdout:25 2024-06-15T10:28:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-06-15T10:28:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:28:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:28:56.140 INFO:tasks.workunit.client.0.smithi116.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2678939831,v1:127.0.0.1:6875/2678939831] [v2:127.0.0.1:6876/2678939831,v1:127.0.0.1:6877/2678939831] exists,up 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:28:56.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:28:56.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:28:56.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:28:56.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-15T10:28:56.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:28:56.372 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:28:56.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:28:56.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:28:56.969 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:28:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:28:57.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:29:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-15T10:29:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:29:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:29:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:29:02.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:29:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:29:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:29:02.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:29:02.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:29:02.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:29:02.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:29:03.084 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:29:03.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-06-15T10:29:03.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-06-15T10:29:03.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-06-15T10:29:03.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:29:03.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-06-15T10:29:03.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-06-15T10:29:03.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038' 2024-06-15T10:29:03.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:29:03.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-06-15T10:29:03.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-06-15T10:29:03.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510' 2024-06-15T10:29:03.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:29:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345983 2024-06-15T10:29:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345983 2024-06-15T10:29:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983' 2024-06-15T10:29:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:29:03.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-06-15T10:29:03.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-06-15T10:29:03.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448' 2024-06-15T10:29:03.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:03.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:29:04.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-06-15T10:29:04.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-06-15T10:29:04.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448 5-128849018924' 2024-06-15T10:29:04.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:04.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:29:04.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855392 2024-06-15T10:29:04.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855392 2024-06-15T10:29:04.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448 5-128849018924 6-150323855392' 2024-06-15T10:29:04.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:04.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:29:04.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-06-15T10:29:04.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-06-15T10:29:04.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448 5-128849018924 6-150323855392 7-171798691865' 2024-06-15T10:29:04.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:04.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:29:04.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-06-15T10:29:04.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-06-15T10:29:04.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448 5-128849018924 6-150323855392 7-171798691865 8-193273528340' 2024-06-15T10:29:04.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:04.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:29:04.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364809 2024-06-15T10:29:04.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364809 2024-06-15T10:29:04.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509510 3-85899345983 4-107374182448 5-128849018924 6-150323855392 7-171798691865 8-193273528340 9-214748364809' 2024-06-15T10:29:04.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:04.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-06-15T10:29:04.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:04.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:29:04.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-06-15T10:29:04.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:04.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-06-15T10:29:04.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-06-15T10:29:04.636 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836564 2024-06-15T10:29:04.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:29:04.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836564 2024-06-15T10:29:04.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:04.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-06-15T10:29:04.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:04.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:29:04.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-06-15T10:29:04.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:04.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-06-15T10:29:04.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-06-15T10:29:04.873 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673038 2024-06-15T10:29:04.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:29:05.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673038 2024-06-15T10:29:05.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:05.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-06-15T10:29:05.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:05.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:29:05.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-06-15T10:29:05.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:05.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-06-15T10:29:05.160 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509510 2024-06-15T10:29:05.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-06-15T10:29:05.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:29:05.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-06-15T10:29:05.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:05.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345983 2024-06-15T10:29:05.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:05.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:29:05.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345983 2024-06-15T10:29:05.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:05.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345983 2024-06-15T10:29:05.447 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345983 2024-06-15T10:29:05.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345983' 2024-06-15T10:29:05.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:29:05.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345983 -lt 85899345983 2024-06-15T10:29:05.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:05.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-06-15T10:29:05.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:29:05.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-06-15T10:29:05.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:05.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-06-15T10:29:05.686 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182448 2024-06-15T10:29:05.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-06-15T10:29:05.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:29:05.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-06-15T10:29:05.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:05.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-06-15T10:29:05.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:05.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:29:05.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-06-15T10:29:05.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:05.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-06-15T10:29:05.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-06-15T10:29:05.940 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.5 seq 128849018924 2024-06-15T10:29:05.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:29:06.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018924 2024-06-15T10:29:06.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:06.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855392 2024-06-15T10:29:06.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:29:06.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855392 2024-06-15T10:29:06.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:06.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855392 2024-06-15T10:29:06.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-06-15T10:29:06.215 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.6 seq 150323855392 2024-06-15T10:29:06.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:29:06.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855392 -lt 150323855392 2024-06-15T10:29:06.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:06.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-06-15T10:29:06.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:06.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:29:06.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-06-15T10:29:06.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:06.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-06-15T10:29:06.484 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.7 seq 171798691865 2024-06-15T10:29:06.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-06-15T10:29:06.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:29:06.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691866 -lt 171798691865 2024-06-15T10:29:06.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:06.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-06-15T10:29:06.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:06.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:29:06.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-06-15T10:29:06.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-06-15T10:29:06.764 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.8 seq 193273528340 2024-06-15T10:29:06.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-06-15T10:29:06.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:29:07.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-06-15T10:29:07.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:07.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364809 2024-06-15T10:29:07.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:07.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:29:07.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364809 2024-06-15T10:29:07.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:07.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364809 2024-06-15T10:29:07.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-06-15T10:29:07.042 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.9 seq 214748364809 2024-06-15T10:29:07.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:29:07.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364810 -lt 214748364809 2024-06-15T10:29:07.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:29:07.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:07.279 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:07.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:29:07.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:29:07.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:29:07.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:29:07.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:29:07.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:29:07.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:29:07.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:29:07.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:29:07.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:29:07.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:07.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:29:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:29:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:29:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-06-15T10:29:08.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:29:08.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:29:08.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T10:29:08.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:29:08.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-15T10:29:08.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:29:08.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:29:09.614 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:29:09.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:29:10.640 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:29:10.641 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:29:10.642 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:29:10.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:29:10.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:29:10.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:29:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:29:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:29:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:29:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:29:10.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:29:10.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:29:11.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:29:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:11.014 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:29:11.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-06-15T10:29:11.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-06-15T10:29:11.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-06-15T10:29:11.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:11.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:29:11.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-06-15T10:29:11.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-06-15T10:29:11.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041' 2024-06-15T10:29:11.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:11.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:29:11.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-06-15T10:29:11.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-06-15T10:29:11.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512' 2024-06-15T10:29:11.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:11.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:29:11.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345985 2024-06-15T10:29:11.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345985 2024-06-15T10:29:11.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985' 2024-06-15T10:29:11.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:11.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:29:12.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182450 2024-06-15T10:29:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182450 2024-06-15T10:29:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450' 2024-06-15T10:29:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:12.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:29:12.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018926 2024-06-15T10:29:12.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018926 2024-06-15T10:29:12.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450 5-128849018926' 2024-06-15T10:29:12.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:12.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:29:12.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855395 2024-06-15T10:29:12.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855395 2024-06-15T10:29:12.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450 5-128849018926 6-150323855395' 2024-06-15T10:29:12.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:12.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:29:12.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-06-15T10:29:12.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-06-15T10:29:12.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450 5-128849018926 6-150323855395 7-171798691868' 2024-06-15T10:29:12.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:12.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:29:12.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528343 2024-06-15T10:29:12.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528343 2024-06-15T10:29:12.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450 5-128849018926 6-150323855395 7-171798691868 8-193273528343' 2024-06-15T10:29:12.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:12.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:29:13.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-06-15T10:29:13.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-06-15T10:29:13.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509512 3-85899345985 4-107374182450 5-128849018926 6-150323855395 7-171798691868 8-193273528343 9-214748364812' 2024-06-15T10:29:13.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:13.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-06-15T10:29:13.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:13.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:29:13.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-06-15T10:29:13.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:13.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-06-15T10:29:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-06-15T10:29:13.035 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836567 2024-06-15T10:29:13.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:29:13.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836567 2024-06-15T10:29:13.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:13.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-06-15T10:29:13.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:13.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:29:13.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-06-15T10:29:13.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:13.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-06-15T10:29:13.317 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673041 2024-06-15T10:29:13.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-06-15T10:29:13.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:29:13.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-06-15T10:29:13.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:13.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-06-15T10:29:13.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:13.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:29:13.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-06-15T10:29:13.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:13.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-06-15T10:29:13.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-06-15T10:29:13.560 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509512 2024-06-15T10:29:13.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:29:13.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-06-15T10:29:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:13.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345985 2024-06-15T10:29:13.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:13.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:29:13.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345985 2024-06-15T10:29:13.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:13.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345985 2024-06-15T10:29:13.944 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345985 2024-06-15T10:29:13.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345985' 2024-06-15T10:29:13.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:29:14.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345985 -lt 85899345985 2024-06-15T10:29:14.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:14.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182450 2024-06-15T10:29:14.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:14.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:29:14.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182450 2024-06-15T10:29:14.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:14.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182450 2024-06-15T10:29:14.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-06-15T10:29:14.228 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182450 2024-06-15T10:29:14.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:29:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182450 -lt 107374182450 2024-06-15T10:29:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:14.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018926 2024-06-15T10:29:14.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:14.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:29:14.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:14.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018926 2024-06-15T10:29:14.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018926 2024-06-15T10:29:14.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-06-15T10:29:14.467 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.5 seq 128849018926 2024-06-15T10:29:14.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:29:14.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018927 -lt 128849018926 2024-06-15T10:29:14.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:14.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855395 2024-06-15T10:29:14.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:14.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:29:14.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855395 2024-06-15T10:29:14.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:14.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855395 2024-06-15T10:29:14.727 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.6 seq 150323855395 2024-06-15T10:29:14.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-06-15T10:29:14.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:29:15.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855395 -lt 150323855395 2024-06-15T10:29:15.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:15.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-06-15T10:29:15.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:15.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:29:15.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-06-15T10:29:15.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:15.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-06-15T10:29:15.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-06-15T10:29:15.007 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.7 seq 171798691868 2024-06-15T10:29:15.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:29:15.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691868 -lt 171798691868 2024-06-15T10:29:15.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:15.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528343 2024-06-15T10:29:15.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:15.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:29:15.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528343 2024-06-15T10:29:15.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:15.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528343 2024-06-15T10:29:15.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2024-06-15T10:29:15.246 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.8 seq 193273528343 2024-06-15T10:29:15.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:29:15.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528343 -lt 193273528343 2024-06-15T10:29:15.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:15.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:15.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-06-15T10:29:15.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:29:15.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-06-15T10:29:15.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:15.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-06-15T10:29:15.483 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.9 seq 214748364812 2024-06-15T10:29:15.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-06-15T10:29:15.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:29:15.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-06-15T10:29:15.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:29:15.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:15.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:15.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:29:15.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:29:15.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:29:16.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:29:16.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:29:16.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:16.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:29:16.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T10:29:16.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:29:16.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:29:16.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:29:16.833 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:29:16.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:29:17.326 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:29:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:29:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:29:17.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:29:17.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-15T10:29:17.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:29:17.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:29:17.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:29:17.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:29:18.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-15T10:29:18.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-15T10:29:18.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-15T10:29:18.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T10:29:18.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-15T10:29:18.069 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:29:18.070 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:29:18.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-15T10:29:18.343 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:29:18.343 INFO:tasks.workunit.client.0.smithi116.stderr:9 2024-06-15T10:29:18.343 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:29:18.343 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:29:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:18.344 INFO:tasks.workunit.client.0.smithi116.stderr:7' 2024-06-15T10:29:18.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-15T10:29:18.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-15T10:29:18.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-15T10:29:18.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-15T10:29:18.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:29:18.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:29:18.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:29:19.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:29:21.783 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:29:22.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:29:22.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:29:22.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:29:22.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:29:22.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:29:22.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:29:22.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:29:22.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:29:22.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:29:22.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:29:22.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:29:22.334 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:29:22.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:29:22.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:29:22.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:29:22.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:29:22.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:29:22.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:29:22.372 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:22.374+0000 7f7c397167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:22.375 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:22.378+0000 7f7c397167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:22.377 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:22.378+0000 7f7c397167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:29:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:22.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:29:22.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:22.924 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:22.926+0000 7f7c397167c0 -1 Falling back to public interface 2024-06-15T10:29:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:29:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:23.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:29:24.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:25.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:25.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:25.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:29:25.160 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:25.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:25.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:29:25.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:25.961 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:25.962+0000 7f7c397167c0 -1 osd.3 66 log_to_monitors true 2024-06-15T10:29:26.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:26.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:26.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:29:26.474 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:26.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:26.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 70 up_thru 61 down_at 67 last_clean_interval [20,66) [v2:127.0.0.1:6826/3539147820,v1:127.0.0.1:6827/3539147820] [v2:127.0.0.1:6828/3539147820,v1:127.0.0.1:6829/3539147820] exists,up 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:29:26.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:29:26.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:29:27.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:29:27.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:29:27.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:29:27.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:29:27.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:29:27.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:29:27.562 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:29:27.563 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:29:27.563 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:29:27.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:29:27.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:27.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:29:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-06-15T10:29:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-06-15T10:29:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-06-15T10:29:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:27.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:29:28.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-06-15T10:29:28.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-06-15T10:29:28.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045' 2024-06-15T10:29:28.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:28.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:29:28.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-06-15T10:29:28.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-06-15T10:29:28.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517' 2024-06-15T10:29:28.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:28.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:29:28.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-06-15T10:29:28.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-06-15T10:29:28.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722' 2024-06-15T10:29:28.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:28.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:29:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-06-15T10:29:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-06-15T10:29:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455' 2024-06-15T10:29:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:28.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:29:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018931 2024-06-15T10:29:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018931 2024-06-15T10:29:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455 5-128849018931' 2024-06-15T10:29:28.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:28.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:29:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-06-15T10:29:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-06-15T10:29:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455 5-128849018931 6-150323855399' 2024-06-15T10:29:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:29.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:29:29.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691872 2024-06-15T10:29:29.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691872 2024-06-15T10:29:29.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455 5-128849018931 6-150323855399 7-171798691872' 2024-06-15T10:29:29.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:29.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:29:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-06-15T10:29:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-06-15T10:29:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455 5-128849018931 6-150323855399 7-171798691872 8-193273528347' 2024-06-15T10:29:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:29.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:29:29.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364816 2024-06-15T10:29:29.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364816 2024-06-15T10:29:29.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509517 3-300647710722 4-107374182455 5-128849018931 6-150323855399 7-171798691872 8-193273528347 9-214748364816' 2024-06-15T10:29:29.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:29.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-06-15T10:29:29.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:29.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:29:29.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:29.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-06-15T10:29:29.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-06-15T10:29:29.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-06-15T10:29:29.712 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836571 2024-06-15T10:29:29.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:29:29.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-06-15T10:29:29.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:29.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:29.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-06-15T10:29:29.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:29:29.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-06-15T10:29:29.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:29.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-06-15T10:29:29.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-06-15T10:29:29.954 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673045 2024-06-15T10:29:29.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:29:30.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-06-15T10:29:30.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:30.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:30.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-06-15T10:29:30.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:29:30.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-06-15T10:29:30.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:30.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-06-15T10:29:30.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-06-15T10:29:30.233 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509517 2024-06-15T10:29:30.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:29:30.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509517 2024-06-15T10:29:30.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:30.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:30.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710722 2024-06-15T10:29:30.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:29:30.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710722 2024-06-15T10:29:30.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:30.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-06-15T10:29:30.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710722' 2024-06-15T10:29:30.508 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 300647710722 2024-06-15T10:29:30.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:29:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710722 2024-06-15T10:29:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:30.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-06-15T10:29:30.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:29:30.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-06-15T10:29:30.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:30.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-06-15T10:29:30.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-06-15T10:29:30.784 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182455 2024-06-15T10:29:30.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:29:31.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182455 2024-06-15T10:29:31.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:31.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:31.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018931 2024-06-15T10:29:31.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:29:31.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018931 2024-06-15T10:29:31.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:31.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018931 2024-06-15T10:29:31.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018931' 2024-06-15T10:29:31.047 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 128849018931 2024-06-15T10:29:31.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:29:31.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018931 -lt 128849018931 2024-06-15T10:29:31.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:31.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-06-15T10:29:31.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:29:31.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-06-15T10:29:31.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-06-15T10:29:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-06-15T10:29:31.270 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855399 2024-06-15T10:29:31.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:29:31.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855400 -lt 150323855399 2024-06-15T10:29:31.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:31.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691872 2024-06-15T10:29:31.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:31.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:29:31.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:31.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691872 2024-06-15T10:29:31.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691872 2024-06-15T10:29:31.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-06-15T10:29:31.527 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691872 2024-06-15T10:29:31.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:29:31.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691872 -lt 171798691872 2024-06-15T10:29:31.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:31.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-06-15T10:29:31.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:31.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:29:31.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-06-15T10:29:31.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:31.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-06-15T10:29:31.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-06-15T10:29:31.831 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528347 2024-06-15T10:29:31.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:29:32.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528347 2024-06-15T10:29:32.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:32.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364816 2024-06-15T10:29:32.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:32.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:29:32.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364816 2024-06-15T10:29:32.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:32.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364816 2024-06-15T10:29:32.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-06-15T10:29:32.116 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 214748364816 2024-06-15T10:29:32.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:29:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364816 -lt 214748364816 2024-06-15T10:29:32.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:29:32.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:32.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:32.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:29:32.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:29:32.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:29:32.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:29:32.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:29:32.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:29:32.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:29:32.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:29:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:29:33.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:29:33.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:33.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:29:33.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:29:33.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:29:33.351 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:29:33.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:29:33.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:29:33.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:29:33.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:29:33.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:29:33.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:29:33.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:29:33.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:29:33.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:33.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:29:34.143 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:29:34.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:29:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:29:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:29:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:29:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:29:34.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:29:34.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:09.450180+0000 '>' 2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:34.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:29:35.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:29:35.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:29:35.397 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:29:35.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:29:35.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:29:35.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:29:35.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:29:35.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:09.450180+0000 '>' 2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:35.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:29:36.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:29:36.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:29:36.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:29:36.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:29:36.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:29:36.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:29:36.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:29:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:34.961547+0000 '>' 2024-06-15T10:29:09.450180+0000 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:29:36.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:29:36.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:29:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:29:39.865 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:29:39.865 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:29:39.865 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:29:40.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:29:40.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:29:40.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:29:40.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:29:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:29:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.9 2024-06-15T10:29:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:29:40.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:29:40.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:29:40.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:29:40.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:29:40.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:29:40.192 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:40.194+0000 7fc8acddd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:40.195 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:40.198+0000 7fc8acddd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:40.196 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:40.198+0000 7fc8acddd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:29:40.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:40.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:29:40.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:40.734 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:40.734+0000 7fc8acddd7c0 -1 Falling back to public interface 2024-06-15T10:29:41.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:41.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:41.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:29:41.652 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:41.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:41.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:29:41.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:42.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:42.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:42.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:29:42.893 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:42.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:42.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:29:43.127 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:43.130+0000 7fc8acddd7c0 -1 osd.9 71 log_to_monitors true 2024-06-15T10:29:43.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:44.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:29:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:44.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:29:44.622 INFO:tasks.workunit.client.0.smithi116.stderr:osd.9 up in weight 1 up_from 75 up_thru 0 down_at 72 last_clean_interval [50,71) [v2:127.0.0.1:6874/546112016,v1:127.0.0.1:6875/546112016] [v2:127.0.0.1:6876/546112016,v1:127.0.0.1:6877/546112016] exists,up 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:29:44.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:29:44.624 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:29:44.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:29:44.625 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:29:44.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:29:44.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:29:44.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:29:44.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:29:44.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:29:44.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:29:44.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:29:44.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:29:44.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:29:45.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:29:45.302 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:45.302 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:45.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:29:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-06-15T10:29:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-06-15T10:29:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-06-15T10:29:45.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:45.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:29:45.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-06-15T10:29:45.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-06-15T10:29:45.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050' 2024-06-15T10:29:45.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:45.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:29:46.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-06-15T10:29:46.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-06-15T10:29:46.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521' 2024-06-15T10:29:46.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:46.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:29:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710727 2024-06-15T10:29:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710727 2024-06-15T10:29:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727' 2024-06-15T10:29:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:46.332 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:29:46.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182459 2024-06-15T10:29:46.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182459 2024-06-15T10:29:46.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459' 2024-06-15T10:29:46.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:46.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:29:47.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018935 2024-06-15T10:29:47.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018935 2024-06-15T10:29:47.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459 5-128849018935' 2024-06-15T10:29:47.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:47.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:29:47.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-06-15T10:29:47.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-06-15T10:29:47.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459 5-128849018935 6-150323855404' 2024-06-15T10:29:47.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:47.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:29:47.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-06-15T10:29:47.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-06-15T10:29:47.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459 5-128849018935 6-150323855404 7-171798691877' 2024-06-15T10:29:47.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:47.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:29:47.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528352 2024-06-15T10:29:47.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528352 2024-06-15T10:29:47.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459 5-128849018935 6-150323855404 7-171798691877 8-193273528352' 2024-06-15T10:29:47.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:29:47.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:29:47.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-06-15T10:29:47.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-06-15T10:29:47.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509521 3-300647710727 4-107374182459 5-128849018935 6-150323855404 7-171798691877 8-193273528352 9-322122547202' 2024-06-15T10:29:47.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:47.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-06-15T10:29:47.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:47.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:29:47.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-06-15T10:29:47.615 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:47.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-06-15T10:29:47.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-06-15T10:29:47.616 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836576 2024-06-15T10:29:47.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:29:47.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-06-15T10:29:47.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:47.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-06-15T10:29:47.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:47.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:29:47.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-06-15T10:29:47.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-06-15T10:29:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-06-15T10:29:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673050 2024-06-15T10:29:47.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:29:48.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-06-15T10:29:48.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:48.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-06-15T10:29:48.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:48.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:29:48.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-06-15T10:29:48.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:48.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-06-15T10:29:48.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-06-15T10:29:48.081 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509521 2024-06-15T10:29:48.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:29:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509521 2024-06-15T10:29:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:48.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:48.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710727 2024-06-15T10:29:48.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:29:48.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710727 2024-06-15T10:29:48.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:48.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710727 2024-06-15T10:29:48.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710727' 2024-06-15T10:29:48.325 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 300647710727 2024-06-15T10:29:48.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:29:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710727 -lt 300647710727 2024-06-15T10:29:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:48.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:48.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182459 2024-06-15T10:29:48.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:29:48.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182459 2024-06-15T10:29:48.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:48.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182459 2024-06-15T10:29:48.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-06-15T10:29:48.549 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182459 2024-06-15T10:29:48.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:29:48.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182459 2024-06-15T10:29:48.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:48.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018935 2024-06-15T10:29:48.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:48.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:29:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018935 2024-06-15T10:29:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:48.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018935 2024-06-15T10:29:48.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018935' 2024-06-15T10:29:48.802 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 128849018935 2024-06-15T10:29:48.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:29:49.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018936 -lt 128849018935 2024-06-15T10:29:49.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:49.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-06-15T10:29:49.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:49.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:29:49.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-06-15T10:29:49.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-06-15T10:29:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-06-15T10:29:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855404 2024-06-15T10:29:49.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:29:49.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855404 2024-06-15T10:29:49.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:49.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:49.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-06-15T10:29:49.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:29:49.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-06-15T10:29:49.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:49.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-06-15T10:29:49.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-06-15T10:29:49.375 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691877 2024-06-15T10:29:49.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:29:49.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691877 2024-06-15T10:29:49.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:49.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:49.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528352 2024-06-15T10:29:49.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:29:49.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528352 2024-06-15T10:29:49.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:49.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528352 2024-06-15T10:29:49.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-06-15T10:29:49.648 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528352 2024-06-15T10:29:49.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:29:49.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528352 2024-06-15T10:29:49.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:29:49.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:29:49.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547202 2024-06-15T10:29:49.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:29:49.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547202 2024-06-15T10:29:49.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:29:49.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-06-15T10:29:49.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547202' 2024-06-15T10:29:49.931 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 322122547202 2024-06-15T10:29:49.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:29:50.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547202 -lt 322122547202 2024-06-15T10:29:50.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:29:50.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:50.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:50.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:29:50.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:29:50.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:29:50.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:29:50.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:29:50.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:29:50.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:29:50.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:29:50.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:29:50.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:29:50.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:29:50.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:29:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:29:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:29:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:29:51.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:29:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:29:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:29:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:29:51.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:29:51.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:29:54.700 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:29:55.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:29:55.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:29:55.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:29:55.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:29:55.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:29:55.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:29:55.244 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:29:55.244 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:29:55.244 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:29:55.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:29:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.5 2024-06-15T10:29:55.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:29:55.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:29:55.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:29:55.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:29:55.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:29:55.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:29:55.279 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:55.281+0000 7f6fd68c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:55.320 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:55.321+0000 7f6fd68c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:55.322 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:55.321+0000 7f6fd68c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:29:55.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:55.857+0000 7f6fd68c47c0 -1 Falling back to public interface 2024-06-15T10:29:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:29:56.740 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:29:56.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:56.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:29:56.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:57.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:57.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:57.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:29:57.984 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:29:57.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:57.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:29:58.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:29:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:29:58.917+0000 7f6fd68c47c0 -1 osd.5 76 log_to_monitors true 2024-06-15T10:29:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:29:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:29:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:29:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:29:59.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:29:59.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:29:59.637 INFO:tasks.workunit.client.0.smithi116.stderr:osd.5 up in weight 1 up_from 81 up_thru 67 down_at 77 last_clean_interval [30,76) [v2:127.0.0.1:6842/3593560703,v1:127.0.0.1:6843/3593560703] [v2:127.0.0.1:6844/3593560703,v1:127.0.0.1:6845/3593560703] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:29:59.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:29:59.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:29:59.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:29:59.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:29:59.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:29:59.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:29:59.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:29:59.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:29:59.640 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:29:59.641 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:29:59.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:29:59.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:29:59.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:29:59.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:29:59.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:29:59.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:29:59.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:29:59.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:29:59.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:30:00.187 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:30:00.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:00.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:00.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:00.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-06-15T10:30:00.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-06-15T10:30:00.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-06-15T10:30:00.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:00.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:30:01.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-06-15T10:30:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-06-15T10:30:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054' 2024-06-15T10:30:01.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:01.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:30:01.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-06-15T10:30:01.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-06-15T10:30:01.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525' 2024-06-15T10:30:01.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:01.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:30:01.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710731 2024-06-15T10:30:01.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710731 2024-06-15T10:30:01.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731' 2024-06-15T10:30:01.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:01.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:30:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-06-15T10:30:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-06-15T10:30:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463' 2024-06-15T10:30:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:01.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:30:01.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-06-15T10:30:01.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-06-15T10:30:01.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463 5-347892350978' 2024-06-15T10:30:01.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:01.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:30:02.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-06-15T10:30:02.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-06-15T10:30:02.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463 5-347892350978 6-150323855408' 2024-06-15T10:30:02.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:02.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:30:02.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691881 2024-06-15T10:30:02.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691881 2024-06-15T10:30:02.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463 5-347892350978 6-150323855408 7-171798691881' 2024-06-15T10:30:02.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:02.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:30:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-06-15T10:30:02.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-06-15T10:30:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463 5-347892350978 6-150323855408 7-171798691881 8-193273528356' 2024-06-15T10:30:02.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:02.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:30:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547206 2024-06-15T10:30:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547206 2024-06-15T10:30:02.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673054 2-64424509525 3-300647710731 4-107374182463 5-347892350978 6-150323855408 7-171798691881 8-193273528356 9-322122547206' 2024-06-15T10:30:02.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:02.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-06-15T10:30:02.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:02.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:30:02.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-06-15T10:30:02.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:02.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-06-15T10:30:02.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-06-15T10:30:02.819 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836580 2024-06-15T10:30:02.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:30:03.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836580 2024-06-15T10:30:03.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:03.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-06-15T10:30:03.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:03.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:30:03.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-06-15T10:30:03.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:03.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-06-15T10:30:03.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-06-15T10:30:03.056 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673054 2024-06-15T10:30:03.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:30:03.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-06-15T10:30:03.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:03.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-06-15T10:30:03.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:03.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:30:03.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-06-15T10:30:03.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:03.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-06-15T10:30:03.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-06-15T10:30:03.334 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509525 2024-06-15T10:30:03.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:30:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-06-15T10:30:03.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:03.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710731 2024-06-15T10:30:03.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:03.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:30:03.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710731 2024-06-15T10:30:03.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710731 2024-06-15T10:30:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710731' 2024-06-15T10:30:03.599 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 300647710731 2024-06-15T10:30:03.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:30:04.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710731 -lt 300647710731 2024-06-15T10:30:04.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:04.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-06-15T10:30:04.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:04.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:30:04.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-06-15T10:30:04.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-06-15T10:30:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-06-15T10:30:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182463 2024-06-15T10:30:04.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:30:04.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182463 2024-06-15T10:30:04.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:04.375 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350978 2024-06-15T10:30:04.375 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:04.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:30:04.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350978 2024-06-15T10:30:04.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:04.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-06-15T10:30:04.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350978' 2024-06-15T10:30:04.380 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 347892350978 2024-06-15T10:30:04.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:30:04.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350978 2024-06-15T10:30:04.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:04.689 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-06-15T10:30:04.689 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:04.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:30:04.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-06-15T10:30:04.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:04.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-06-15T10:30:04.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-06-15T10:30:04.694 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855408 2024-06-15T10:30:04.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:30:04.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855408 -lt 150323855408 2024-06-15T10:30:04.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:04.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691881 2024-06-15T10:30:04.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:04.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:30:04.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691881 2024-06-15T10:30:04.930 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:04.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691881 2024-06-15T10:30:04.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-06-15T10:30:04.932 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691881 2024-06-15T10:30:04.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:30:05.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691881 -lt 171798691881 2024-06-15T10:30:05.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:05.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-06-15T10:30:05.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:05.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:30:05.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-06-15T10:30:05.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:05.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-06-15T10:30:05.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-06-15T10:30:05.202 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528356 2024-06-15T10:30:05.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:30:05.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528356 2024-06-15T10:30:05.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:05.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547206 2024-06-15T10:30:05.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:05.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:30:05.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547206 2024-06-15T10:30:05.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:05.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547206 2024-06-15T10:30:05.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547206' 2024-06-15T10:30:05.465 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 322122547206 2024-06-15T10:30:05.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:30:05.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547206 2024-06-15T10:30:05.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:30:05.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:05.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:05.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:30:05.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:30:05.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:30:05.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:30:05.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:30:05.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:30:05.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:30:05.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:30:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:30:06.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:30:06.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:06.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:06.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:30:06.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:30:06.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:30:06.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:30:06.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:30:06.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:30:06.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:30:06.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:30:06.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:30:06.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:30:06.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:30:06.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:30:06.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:06.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:06.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:06.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:07.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:07.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:30:07.226 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:30:07.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:07.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:07.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:07.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:07.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:34.961547+0000 '>' 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:07.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:08.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:08.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:08.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:08.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:08.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:08.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:08.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:08.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:34.961547+0000 '>' 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:08.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:09.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:09.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:09.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:09.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:09.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:09.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:09.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:09.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:34.961547+0000 '>' 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:09.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:10.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:10.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:10.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:11.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:29:34.961547+0000 '>' 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:11.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:12.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:12.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:12.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:12.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:12.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:12.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:12.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:07.557300+0000 '>' 2024-06-15T10:29:34.961547+0000 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:30:12.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:30:12.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:30:12.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:30:12.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:12.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:30:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:30:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:30:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:30:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:12.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:30:15.560 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:30:15.560 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:30:15.560 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:30:15.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:30:15.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:30:15.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:30:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:30:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:30:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:30:15.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:30:15.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:30:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:30:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.9 2024-06-15T10:30:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:30:15.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:30:15.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:30:15.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:30:15.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:30:15.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:30:15.905 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:30:15.905+0000 7f87afe0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:15.907 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:30:15.909+0000 7f87afe0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:15.909 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:30:15.909+0000 7f87afe0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:16.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:30:16.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:16.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:16.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:16.464 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:30:16.465+0000 7f87afe0f7c0 -1 Falling back to public interface 2024-06-15T10:30:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:30:17.412 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:30:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:17.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:18.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:18.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:30:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:30:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:18.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:18.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:18.901 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:30:18.901+0000 7f87afe0f7c0 -1 osd.9 82 log_to_monitors true 2024-06-15T10:30:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:30:19.902 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:30:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:19.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:20.161 INFO:tasks.workunit.client.0.smithi116.stderr:osd.9 up in weight 1 up_from 87 up_thru 0 down_at 83 last_clean_interval [75,82) [v2:127.0.0.1:6874/2962476935,v1:127.0.0.1:6875/2962476935] [v2:127.0.0.1:6876/2962476935,v1:127.0.0.1:6877/2962476935] exists,up 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:30:20.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:30:20.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:30:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:30:20.163 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:30:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:30:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:30:20.164 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:30:20.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:30:20.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:30:20.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:30:20.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:30:20.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:30:20.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:30:20.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:20.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:20.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:21.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-06-15T10:30:21.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-06-15T10:30:21.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-06-15T10:30:21.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:21.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:30:21.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-06-15T10:30:21.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-06-15T10:30:21.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059' 2024-06-15T10:30:21.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:21.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:30:21.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-06-15T10:30:21.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-06-15T10:30:21.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530' 2024-06-15T10:30:21.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:21.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:30:22.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710736 2024-06-15T10:30:22.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710736 2024-06-15T10:30:22.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736' 2024-06-15T10:30:22.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:22.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:30:22.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-06-15T10:30:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-06-15T10:30:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468' 2024-06-15T10:30:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:22.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:30:22.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350983 2024-06-15T10:30:22.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350983 2024-06-15T10:30:22.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468 5-347892350983' 2024-06-15T10:30:22.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:22.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:30:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-06-15T10:30:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-06-15T10:30:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468 5-347892350983 6-150323855413' 2024-06-15T10:30:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:22.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:30:22.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-06-15T10:30:22.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-06-15T10:30:22.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468 5-347892350983 6-150323855413 7-171798691886' 2024-06-15T10:30:22.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:22.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:30:23.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-06-15T10:30:23.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-06-15T10:30:23.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468 5-347892350983 6-150323855413 7-171798691886 8-193273528361' 2024-06-15T10:30:23.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:23.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:30:23.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-06-15T10:30:23.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-06-15T10:30:23.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509530 3-300647710736 4-107374182468 5-347892350983 6-150323855413 7-171798691886 8-193273528361 9-373662154754' 2024-06-15T10:30:23.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:23.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-06-15T10:30:23.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:23.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:30:23.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-06-15T10:30:23.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:23.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-06-15T10:30:23.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-06-15T10:30:23.240 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836585 2024-06-15T10:30:23.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:30:23.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836585 2024-06-15T10:30:23.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:23.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-06-15T10:30:23.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:23.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:30:23.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-06-15T10:30:23.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:23.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-06-15T10:30:23.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-06-15T10:30:23.474 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673059 2024-06-15T10:30:23.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:30:23.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-06-15T10:30:23.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:23.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-06-15T10:30:23.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:23.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:30:23.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-06-15T10:30:23.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-06-15T10:30:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-06-15T10:30:23.760 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509530 2024-06-15T10:30:23.761 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:30:24.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-06-15T10:30:24.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:24.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710736 2024-06-15T10:30:24.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:24.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:30:24.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710736 2024-06-15T10:30:24.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:24.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710736 2024-06-15T10:30:24.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710736' 2024-06-15T10:30:24.041 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 300647710736 2024-06-15T10:30:24.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:30:24.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710736 -lt 300647710736 2024-06-15T10:30:24.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:24.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-06-15T10:30:24.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:24.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:30:24.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-06-15T10:30:24.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:24.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-06-15T10:30:24.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-06-15T10:30:24.344 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182468 2024-06-15T10:30:24.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:30:24.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182468 2024-06-15T10:30:24.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:24.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350983 2024-06-15T10:30:24.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:24.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:30:24.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350983 2024-06-15T10:30:24.583 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350983 2024-06-15T10:30:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350983' 2024-06-15T10:30:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 347892350983 2024-06-15T10:30:24.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:30:24.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350984 -lt 347892350983 2024-06-15T10:30:24.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:24.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-06-15T10:30:24.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:24.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:30:24.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-06-15T10:30:24.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:24.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-06-15T10:30:24.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-06-15T10:30:24.818 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855413 2024-06-15T10:30:24.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:30:25.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855413 2024-06-15T10:30:25.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:25.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-06-15T10:30:25.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:25.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:30:25.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-06-15T10:30:25.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-06-15T10:30:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-06-15T10:30:25.077 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691886 2024-06-15T10:30:25.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:30:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691886 2024-06-15T10:30:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:25.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-06-15T10:30:25.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:25.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:30:25.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-06-15T10:30:25.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-06-15T10:30:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-06-15T10:30:25.365 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528361 2024-06-15T10:30:25.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:30:25.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528361 -lt 193273528361 2024-06-15T10:30:25.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:25.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-373662154754 2024-06-15T10:30:25.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:25.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:30:25.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:25.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-373662154754 2024-06-15T10:30:25.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-06-15T10:30:25.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 373662154754' 2024-06-15T10:30:25.636 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 373662154754 2024-06-15T10:30:25.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:30:25.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154754 -lt 373662154754 2024-06-15T10:30:25.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:30:25.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:25.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:26.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:30:26.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:30:26.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:30:26.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:30:26.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:26.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:26.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:30:26.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:30:26.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:30:26.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:30:26.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:30:26.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-15T10:30:26.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:30:26.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:30:26.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229394"' 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229394' 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229396"' 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:30:26.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229396' 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229394 229396' 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229397: /' 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229394 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:30:26.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229399: /' 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:30:30.758 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:30:30.759 INFO:tasks.workunit.client.0.smithi116.stderr:229397: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:30:30.760 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:30:30.762 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:30:30.763 INFO:tasks.workunit.client.0.smithi116.stderr:229397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:30:30.764 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:30:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:229397: start osd.5 2024-06-15T10:30:30.765 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:30:30.843 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.843 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:30:30.843 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:30:30.844 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:30:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:30:30.846 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:30:30.846 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:30.846 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:30:30.846 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:30:30.847 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:30:30.847 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:30:30.847 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: start osd.9 2024-06-15T10:30:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:30:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2024-06-15T10:30:30.877+0000 7f99857307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2024-06-15T10:30:30.889+0000 7f99857307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2024-06-15T10:30:30.889+0000 7f99857307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 0 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2024-06-15T10:30:31.441+0000 7f99857307c0 -1 Falling back to public interface 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 1 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2 2024-06-15T10:30:35.469 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2024-06-15T10:30:34.517+0000 7f99857307c0 -1 osd.9 89 log_to_monitors true 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 3 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:35.470 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.500 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2024-06-15T10:30:30.801+0000 7f8a6b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2024-06-15T10:30:30.889+0000 7f8a6b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2024-06-15T10:30:30.889+0000 7f8a6b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:30:35.501 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 0 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2024-06-15T10:30:31.449+0000 7f8a6b6127c0 -1 Falling back to public interface 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:30:35.502 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 1 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:35.503 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:30:35.504 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 3 2024-06-15T10:30:35.504 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:35.504 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:35.504 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2024-06-15T10:30:35.501+0000 7f8a6b6127c0 -1 osd.5 89 log_to_monitors true 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 4 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: osd.5 up in weight 1 up_from 94 up_thru 81 down_at 90 last_clean_interval [81,89) [v2:127.0.0.1:6874/656172486,v1:127.0.0.1:6875/656172486] [v2:127.0.0.1:6876/656172486,v1:127.0.0.1:6877/656172486] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:30:38.136 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:30:38.137 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 1 2024-06-15T10:30:38.138 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 2 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 3 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 4 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 5 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 6 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 7 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 8 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: 9' 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:38.139 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //hos.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 4 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: osd.9 up in weight 1 up_from 93 up_thru 0 down_at 90 last_clean_interval [87,89) [v2:127.0.0.1:6842/3937865607,v1:127.0.0.1:6843/3937865607] [v2:127.0.0.1:6844/3937865607,v1:127.0.0.1:6845/3937865607] exists,up 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:30:38.140 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:30:38.141 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 1 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 2 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 3 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 4 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 5 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 6 2024-06-15T10:30:38.142 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 7 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 8 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: 9' 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-06-15T10:30:38.143 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.369 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063' 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535' 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.370 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710740 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710740 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740' 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473' 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-06-15T10:30:40.371 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473 5-403726925826' 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473 5-403726925826 6-150323855419' 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.372 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064' 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536' 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.374 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710741 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710741 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741' 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474' 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-06-15T10:30:40.375 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474 5-403726925827' 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474 5-403726925827 6-150323855418' 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.376 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:30:40.916 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691891 2024-06-15T10:30:40.916 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691891 2024-06-15T10:30:40.916 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474 5-403726925827 6-150323855418 7-171798691891' 2024-06-15T10:30:40.916 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.916 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474 5-403726925827 6-150323855418 7-171798691891 8-193273528366' 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509536 3-300647710741 4-107374182474 5-403726925827 6-150323855418 7-171798691891 8-193273528366 9-399431958531' 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:30:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.0 seq 21474836589 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836589 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-06-15T10:30:40.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:40.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-06-15T10:30:40.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-06-15T10:30:40.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.1 seq 42949673064 2024-06-15T10:30:40.923 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_statpg_stats: seq=171798691890 2024-06-15T10:30:40.923 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-06-15T10:30:40.923 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473 5-403726925826 6-150323855419 7-171798691890' 2024-06-15T10:30:40.923 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528365 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528365 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473 5-403726925826 6-150323855419 7-171798691890 8-193273528365' 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958532 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958532 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-300647710740 4-107374182473 5-403726925826 6-150323855419 7-171798691890 8-193273528365 9-399431958532' 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-06-15T10:30:40.924 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.0 seq 21474836590 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836590 2024-06-15T10:30:40.925 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-06-15T10:30:40.926 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.1 seq 42949673063 2024-06-15T10:30:41.906 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673063 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:30:41.907 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-06-15T10:30:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-06-15T10:30:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-06-15T10:30:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.2 seq 64424509535 2024-06-15T10:30:41.908 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509535 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710740 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:30:41.909 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710740 2024-06-15T10:30:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710740 2024-06-15T10:30:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710740' 2024-06-15T10:30:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.3 seq 300647710740 2024-06-15T10:30:41.910 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710741 -lt 300647710740 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:30:41.911 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-06-15T10:30:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-06-15T10:30:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-06-15T10:30:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.4 seq 107374182473 2024-06-15T10:30:41.912 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:30:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182473 2024-06-15T10:30:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.913 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925826 2024-06-15T10:30:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/ces: ceph osd last-stat-seq 1 2024-06-15T10:30:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673064 2024-06-15T10:30:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-06-15T10:30:41.914 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:30:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-06-15T10:30:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-06-15T10:30:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-06-15T10:30:41.917 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.2 seq 64424509536 2024-06-15T10:30:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:30:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-06-15T10:30:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710741 2024-06-15T10:30:41.918 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:30:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710741 2024-06-15T10:30:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710741 2024-06-15T10:30:41.919 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710741' 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.3 seq 300647710741 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710741 -lt 300647710741 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-06-15T10:30:41.920 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:30:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-06-15T10:30:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-06-15T10:30:41.921 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-06-15T10:30:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.4 seq 107374182474 2024-06-15T10:30:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:30:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182474 2024-06-15T10:30:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:41.922 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925827 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925827 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 403726925827' 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.5 seq 403726925827 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-06-15T10:30:42.623 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.6 seq 150323855418 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855418 2024-06-15T10:30:42.624 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691891 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691891 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691891 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.7 seq 171798691891 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691891 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.625 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-06-15T10:30:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:30:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-06-15T10:30:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.626 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925826 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 403726925826' 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.5 seq 403726925826 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:30:42.627 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.6 seq 150323855419 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855419 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-06-15T10:30:42.628 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.7 seq 171798691890 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691892 -lt 171798691890 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528365 2024-06-15T10:30:42.629 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:42.630 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:30:42.630 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528365 2024-06-15T10:30:43.854 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: c-d - -f 2 2024-06-15T10:30:43.854 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-06-15T10:30:43.854 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.8 seq 193273528366 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528366 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-399431958531 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-399431958531 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 399431958531' 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: waiting osd.9 seq 399431958531 2024-06-15T10:30:43.855 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958531 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:30:43.856 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:229397: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:30:43.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229396 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:ut -d - -f 2 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528365 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.8 seq 193273528365 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528365 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-399431958532 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:30:43.883 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-399431958532 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958532 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 399431958532' 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: waiting osd.9 seq 399431958532 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958532 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:30:43.884 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:30:43.885 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:30:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:30:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:229399: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:30:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:30:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:30:43.886 INFO:tasks.workunit.client.0.smithi116.stderr:229399: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:30:43.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:30:43.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:30:43.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:30:43.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:30:43.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:30:43.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:30:44.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:30:44.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:30:44.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:30:44.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:30:44.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:44.112 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:44.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:44.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:44.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:44.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:30:44.584 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:30:44.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:44.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:44.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:44.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:44.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:44.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:44.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:07.557300+0000 '>' 2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:44.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:45.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:45.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:45.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:45.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:45.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:45.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:45.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:46.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:07.557300+0000 '>' 2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:46.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:30:47.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:30:47.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:30:47.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:30:47.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:30:47.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:30:47.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:30:47.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:07.557300+0000 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232798"' 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232798' 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:30:47.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232799"' 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232799' 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 232798 232799' 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232798 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232801: /' 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:47.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232803: /' 2024-06-15T10:30:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:30:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:30:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:30:51.444 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:30:51.445 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: _ 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: hinfo_key 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: snapset 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:30:51.446 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:30:51.447 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:30:51.449 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:30:51.450 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:30:51.451 INFO:tasks.workunit.client.0.smithi116.stderr:232801: start osd.5 2024-06-15T10:30:51.959 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:30:51.960 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: _ 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: hinfo_key 2024-06-15T10:30:51.961 INFO:tasks.workunit.client.0.smithi116.stderr:232803: snapset 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:30:51.962 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:30:51.963 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:30:51.963 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:30:51.963 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:30:51.964 INFO:tasks.workunit.client.0.smithi116.stderr:232803: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:30:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:30:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:30:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:30:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:30:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:30:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:232803: start osd.9 2024-06-15T10:30:56.026 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:30:56.026 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:30:56.026 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2024-06-15T10:30:51.480+0000 7f56c30477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2024-06-15T10:30:51.484+0000 7f56c30477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2024-06-15T10:30:51.488+0000 7f56c30477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.027 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:30:56.028 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 0 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2024-06-15T10:30:52.056+0000 7f56c30477c0 -1 Falling back to public interface 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:30:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 1 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2024-06-15T10:30:55.240+0000 7f56c30477c0 -1 osd.5 96 log_to_monitors true 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.031 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:30:56.032 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 3 2024-06-15T10:30:56.032 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.032 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2024-06-15T10:30:51.996+0000 7fd0598747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2024-06-15T10:30:51.996+0000 7fd0598747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.590 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2024-06-15T10:30:52.000+0000 7fd0598747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 0 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2024-06-15T10:30:52.552+0000 7fd0598747c0 -1 Falling back to public interface 2024-06-15T10:30:56.591 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 1 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:56.592 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2024-06-15T10:30:55.840+0000 7fd0598747c0 -1 osd.9 96 log_to_monitors true 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 3 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:56.593 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:58.516 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 4 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: osd.5 up in weight 1 up_from 101 up_thru 94 down_at 97 last_clean_interval [94,96) [v2:127.0.0.1:6842/3121471932,v1:127.0.0.1:6843/3121471932] [v2:127.0.0.1:6844/3121471932,v1:127.0.0.1:6845/3121471932] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:30:58.517 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:30:58.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 1 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 2 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 3 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 4 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 5 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 6 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 7 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 8 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: 9' 2024-06-15T10:30:58.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836595 2024-06-15T10:30:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836595 2024-06-15T10:30:58.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595' 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ieph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 4 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:30:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: osd.9 up in weight 1 up_from 102 up_thru 0 down_at 97 last_clean_interval [93,96) [v2:127.0.0.1:6874/1346066232,v1:127.0.0.1:6875/1346066232] [v2:127.0.0.1:6876/1346066232,v1:127.0.0.1:6877/1346066232] exists,up 15e6c038-a7d8-4d2b-a7d1-5aa71c2c38c7 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:30:59.138 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 1 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 2 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 3 2024-06-15T10:30:59.139 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 4 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 5 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 6 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 7 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 8 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: 9' 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836596 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836596 2024-06-15T10:30:59.140 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596' 2024-06-15T10:31:00.518 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $idds 2024-06-15T10:31:00.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:00.518 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673069 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673069 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069' 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509541 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509541 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541' 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710746 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710746 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746' 2024-06-15T10:31:00.519 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479' 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479 5-433791696898' 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-06-15T10:31:00.520 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-06-15T10:31:00.521 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479 5-433791696898 6-150323855424' 2024-06-15T10:31:00.521 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.521 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:00.848 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:s 2024-06-15T10:31:00.848 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673071 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673071 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071' 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509542 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509542 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542' 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710747 2024-06-15T10:31:00.849 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710747 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747' 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480' 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696899 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696899 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480 5-433791696899' 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.850 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-06-15T10:31:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-06-15T10:31:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480 5-433791696899 6-150323855425' 2024-06-15T10:31:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:01.076 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22202: flush_pg_stats: seq=171798691896 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479 5-433791696898 6-150323855424 7-171798691896' 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528371 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528371 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479 5-433791696898 6-150323855424 7-171798691896 8-193273528371' 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664194 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664194 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509541 3-300647710746 4-107374182479 5-433791696898 6-150323855424 7-171798691896 8-193273528371 9-438086664194' 2024-06-15T10:31:01.077 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836595 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836595 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836595 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836595' 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.0 seq 21474836595 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836595 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:01.078 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673069 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673069 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673069 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-06-15T10:31:01.079 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.1 seq 42949673069 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214202: flush_pg_stats: seq=171798691897 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480 5-433791696899 6-150323855425 7-171798691897' 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480 5-433791696899 6-150323855425 7-171798691897 8-193273528372' 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:01.349 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664195 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664195 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673071 2-64424509542 3-300647710747 4-107374182480 5-433791696899 6-150323855425 7-171798691897 8-193273528372 9-438086664195' 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836596 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836596 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836596 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.0 seq 21474836596 2024-06-15T10:31:01.350 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836596 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673071 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673071 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673071 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-06-15T10:31:01.351 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.1 seq 42949673071 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214:: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673069 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509541 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509541 2024-06-15T10:31:02.051 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509541 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.2 seq 64424509541 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509541 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710746 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710746 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710746 2024-06-15T10:31:02.052 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710746' 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.3 seq 300647710746 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710747 -lt 300647710746 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.4 seq 107374182479 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:02.053 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182480 -lt 107374182479 2024-06-15T10:31:02.054 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.054 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-433791696898 2024-06-15T10:31:02.386 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //ho flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673071 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509542 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509542 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509542 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-06-15T10:31:02.387 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.2 seq 64424509542 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509542 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710747 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710747 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710747 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710747' 2024-06-15T10:31:02.388 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.3 seq 300647710747 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710747 -lt 300647710747 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.4 seq 107374182480 2024-06-15T10:31:02.389 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:02.390 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182480 -lt 107374182480 2024-06-15T10:31:02.390 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.390 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-433791696899 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //homme/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-433791696898 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 433791696898' 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.5 seq 433791696898 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.810 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.6 seq 150323855424 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855424 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.811 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.7 seq 171798691896 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691896 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528371 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528371 2024-06-15T10:31:03.175 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:03.175 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:03.175 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-433791696899 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696899 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 433791696899' 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.5 seq 433791696899 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696899 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:03.176 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.6 seq 150323855425 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855425 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-06-15T10:31:03.177 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.7 seq 171798691897 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691897 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:03.178 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_ppg_stats: cut -d - -f 2 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528371 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.8 seq 193273528371 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528371 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664194 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664194 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664194 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: waiting osd.9 seq 438086664194 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664194 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:04.115 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:04.116 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:31:04.117 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/g_stats: cut -d - -f 2 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.8 seq 193273528372 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528372 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:04.460 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664195 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664195 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664195 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664195' 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: waiting osd.9 seq 438086664195 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664195 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:04.461 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:04.463 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:31:04.463 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:31:05.150 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/uubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:31:05.150 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:05.150 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:05.151 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:05.152 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:31:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clonbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:05.526 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:05.527 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:05.528 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:06.620 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clonee.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:06.621 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:31:06.622 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:06.623 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:06.623 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:06.623 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:06.623 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:06.623 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtes.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:06.903 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:06.904 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:07.110 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtestt/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:07.110 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:232801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:31:07.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232799 2024-06-15T10:31:07.400 INFO:tasks.workunit.client.0.smithi116.stderr:/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:232803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:31:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:31:07.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:31:07.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:31:07.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:31:07.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:31:07.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236470"' 2024-06-15T10:31:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236470' 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236472"' 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236472' 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 236470 236472' 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:31:07.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236470 2024-06-15T10:31:07.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236473: /' 2024-06-15T10:31:07.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:31:07.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236475: /' 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:31:11.209 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:11.210 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:31:11.211 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:31:11.212 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:31:11.212 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:31:11.212 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:31:11.212 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:31:11.214 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:31:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:31:11.216 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:31:11.217 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:31:11.217 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:31:11.217 INFO:tasks.workunit.client.0.smithi116.stderr:236475: start osd.5 2024-06-15T10:31:11.217 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.460 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:31:11.461 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:31:11.462 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:31:11.463 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:31:11.464 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: start osd.3 2024-06-15T10:31:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:31:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2024-06-15T10:31:11.248+0000 7fc2778f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2024-06-15T10:31:11.288+0000 7fc2778f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2024-06-15T10:31:11.288+0000 7fc2778f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 0 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:15.797 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2024-06-15T10:31:11.840+0000 7fc2778f57c0 -1 Falling back to public interface 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 1 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2 2024-06-15T10:31:15.798 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2024-06-15T10:31:14.848+0000 7fc2778f57c0 -1 osd.5 104 log_to_monitors true 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 3 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:15.799 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:31:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:31:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:31:16.038 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2024-06-15T10:31:11.492+0000 7fbba065a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2024-06-15T10:31:11.528+0000 7fbba065a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2024-06-15T10:31:11.532+0000 7fbba065a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:31:16.039 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 0 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:16.040 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2024-06-15T10:31:12.072+0000 7fbba065a7c0 -1 Falling back to public interface 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 1 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:16.041 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:31:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2 2024-06-15T10:31:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:16.042 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:31:16.043 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 3 2024-06-15T10:31:16.044 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:16.044 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:16.044 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2024-06-15T10:31:15.684+0000 7fbba065a7c0 -1 osd.3 104 log_to_monitors true 2024-06-15T10:31:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:31:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 4 2024-06-15T10:31:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:18.374 INFO:tasks.workunit.client.0.smithi116.stderr:236475: osd.5 up in weight 1 up_from 108 up_thru 101 down_at 105 last_clean_interval [101,104) [v2:127.0.0.1:6826/3154631418,v1:127.0.0.1:6827/3154631418] [v2:127.0.0.1:6828/3154631418,v1:127.0.0.1:6829/3154631418] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:31:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:31:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:31:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:31:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:31:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:31:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:31:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:31:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:31:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:236475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:31:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:236475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:31:18.377 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:31:18.378 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:31:18.378 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:31:18.378 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:31:18.378 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 1 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 2 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 3 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 4 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 5 2024-06-15T10:31:18.379 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 6 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 7 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 8 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: 9' 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:18.380 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:31:18.381 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836601 2024-06-15T10:31:18.381 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836601 2024-06-15T10:31:18.381 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601' 2024-06-15T10:31:18.381 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:19.541 INFO:tasks.workunit.client.0.smithi116.stderr:23rs.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:19.541 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 4 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 5 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:19.542 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: osd.3 up in weight 1 up_from 111 up_thru 102 down_at 105 last_clean_interval [70,104) [v2:127.0.0.1:6842/833626666,v1:127.0.0.1:6843/833626666] [v2:127.0.0.1:6844/833626666,v1:127.0.0.1:6845/833626666] exists,up 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:31:19.543 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 1 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 2 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 3 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 4 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 5 2024-06-15T10:31:19.544 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 6 2024-06-15T10:31:19.545 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 7 2024-06-15T10:31:19.545 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 8 2024-06-15T10:31:19.545 INFO:tasks.workunit.client.0.smithi116.stderr:236473: 9' 2024-06-15T10:31:19.545 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help6475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673075 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673075 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075' 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509547 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509547 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547' 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.296 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858' 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182485 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182485 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485' 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-06-15T10:31:20.297 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485 5-463856467970' 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485 5-463856467970 6-150323855430' 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:20.298 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691902 2024-06-15T10:31:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691902 2024-06-15T10:31:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485 5-463856467970 6-150323855430 7-171798691902' 2024-06-15T10:31:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.094 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528377 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528377 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485 5-463856467970 6-150323855430 7-171798691902 8-193273528377' 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664200 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664200 2024-06-15T10:31:21.095 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509547 3-476741369858 4-107374182485 5-463856467970 6-150323855430 7-171798691902 8-193273528377 9-438086664200' 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836601 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836601 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836601 2024-06-15T10:31:21.096 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.0 seq 21474836601 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836601 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673075 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:21.097 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673075 2024-06-15T10:31:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673075 2024-06-15T10:31:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-06-15T10:31:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.1 seq 42949673075 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: fluers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836603 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836603 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603' 2024-06-15T10:31:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673077 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673077 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077' 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509548 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509548 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548' 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.108 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859' 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182486 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182486 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486' 2024-06-15T10:31:21.109 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467972 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467972 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486 5-463856467972' 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:21.110 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-06-15T10:31:21.780 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-06-15T10:31:21.780 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486 5-463856467972 6-150323855431' 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486 5-463856467972 6-150323855431 7-171798691904' 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:21.781 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528378 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528378 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486 5-463856467972 6-150323855431 7-171798691904 8-193273528378' 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664201 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664201 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509548 3-476741369859 4-107374182486 5-463856467972 6-150323855431 7-171798691904 8-193273528378 9-438086664201' 2024-06-15T10:31:21.782 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836603 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836603 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836603 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.0 seq 21474836603 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836603 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673077 2024-06-15T10:31:21.783 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.105 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cephsh_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673077 -lt 42949673075 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509547 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509547 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509547 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.2 seq 64424509547 2024-06-15T10:31:22.106 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509548 -lt 64424509547 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369858 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369858 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.3 seq 476741369858 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:22.107 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369858 -lt 476741369858 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182485 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182485 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182485 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.4 seq 107374182485 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182485 2024-06-15T10:31:22.108 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.109 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-463856467970 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-463856467970 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 463856467970' 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.5 seq 463856467970 2024-06-15T10:31:22.925 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.6 seq 150323855430 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:22.926 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855430 -lt 150323855430 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691902 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691902 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691902 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.7 seq 171798691902 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691902 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:22.927 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528377 2024-06-15T10:31:22.928 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:22.928 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:22.928 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528377 2024-06-15T10:31:23.903 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_st-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:23.903 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673077 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673077 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.1 seq 42949673077 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673077 -lt 42949673077 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509548 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:23.904 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509548 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509548 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.2 seq 64424509548 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509548 -lt 64424509548 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369859 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369859 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-06-15T10:31:23.905 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.3 seq 476741369859 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369858 -lt 476741369859 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182486 2024-06-15T10:31:23.906 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:24.188 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /ats: cut -d - -f 2 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528377 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.8 seq 193273528377 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528379 -lt 193273528377 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664200 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664200 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:24.189 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664200 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664200' 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: waiting osd.9 seq 438086664200 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664202 -lt 438086664200 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:24.190 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:31:24.191 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:31:24.682 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubunthome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:24.682 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182486 2024-06-15T10:31:24.682 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182486 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.4 seq 107374182486 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182486 -lt 107374182486 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-463856467972 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-463856467972 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467972 2024-06-15T10:31:24.683 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 463856467972' 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.5 seq 463856467972 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467972 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.6 seq 150323855431 2024-06-15T10:31:24.684 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855431 -lt 150323855431 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-06-15T10:31:24.685 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flusu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:24.771 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:236475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:25.619 INFO:tasks.workunit.client.0.smithi116.stderr:h_pg_stats: seq=171798691904 2024-06-15T10:31:25.619 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.7 seq 171798691904 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691904 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528378 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528378 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:25.621 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528378 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.8 seq 193273528378 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528379 -lt 193273528378 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664201 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664201 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664201 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664201' 2024-06-15T10:31:25.622 INFO:tasks.workunit.client.0.smithi116.stderr:236473: waiting osd.9 seq 438086664201 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664202 -lt 438086664201 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:25.623 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:26.133 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:26.133 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:236473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:26.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:26.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:31:26.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236472 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:31:26.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:31:26.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:31:26.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:31:26.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:31:26.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:31:26.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:26.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:26.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:26.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:26.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:26.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:31:26.934 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:31:26.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:26.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:26.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:26.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:26.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:26.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:27.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:27.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:28.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:28.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:28.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:28.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:29.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:29.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:29.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:29.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:29.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:29.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:29.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:29.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:29.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:30.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:30.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:30.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:30.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:30.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:30.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:30.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:30.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:30.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:31.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:31.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:31.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:32.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:32.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:33.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:33.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:33.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:33.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:33.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:33.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:33.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:30:44.707557+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:33.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:31:34.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:31:34.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:31:34.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:31:34.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:31:34.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:31:34.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:31:34.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:31:34.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:31:27.684359+0000 '>' 2024-06-15T10:30:44.707557+0000 2024-06-15T10:31:34.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 240070"' 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 240070' 2024-06-15T10:31:34.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 240071"' 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 240071' 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:31:34.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 240070 240071' 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 240070 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/240073: /' 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:31:34.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/240075: /' 2024-06-15T10:31:38.953 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:31:38.954 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:31:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: _ 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: hinfo_key 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: snapset 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:31:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:31:38.958 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:31:38.958 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:240073: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:31:38.960 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:31:38.961 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:31:38.961 INFO:tasks.workunit.client.0.smithi116.stderr:240073: start osd.3 2024-06-15T10:31:39.678 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:31:39.678 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:31:39.678 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:31:39.679 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:31:39.680 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:31:39.680 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:39.680 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:39.680 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:39.680 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:31:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:31:39.681 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:31:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:31:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:31:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:31:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-15T10:31:39.682 INFO:tasks.workunit.client.0.smithi116.stderr:240075: _ 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: hinfo_key 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: snapset 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:31:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:31:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:31:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:31:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:31:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:31:39.685 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:31:39.685 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:31:39.685 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:31:39.685 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:31:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:31:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:31:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:31:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:31:39.686 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:39.687 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:31:39.688 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:31:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:31:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:31:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:31:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:31:39.689 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:31:39.690 INFO:tasks.workunit.client.0.smithi116.stderr:240075: start osd.5 2024-06-15T10:31:43.009 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2024-06-15T10:31:38.996+0000 7f724186a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2024-06-15T10:31:39.008+0000 7f724186a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2024-06-15T10:31:39.008+0000 7f724186a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:31:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 0 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2024-06-15T10:31:39.568+0000 7f724186a7c0 -1 Falling back to public interface 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 1 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:43.012 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2024-06-15T10:31:42.580+0000 7f724186a7c0 -1 osd.3 112 log_to_monitors true 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 3 2024-06-15T10:31:43.013 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:44.031 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-06-15T10:31:44.031 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:31:44.031 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2024-06-15T10:31:39.720+0000 7f5d36d507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2024-06-15T10:31:39.724+0000 7f5d36d507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2024-06-15T10:31:39.724+0000 7f5d36d507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:31:44.032 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 0 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2024-06-15T10:31:40.260+0000 7f5d36d507c0 -1 Falling back to public interface 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 1 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:44.033 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 3 2024-06-15T10:31:44.034 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:44.035 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:44.035 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2024-06-15T10:31:43.928+0000 7f5d36d507c0 -1 osd.5 112 log_to_monitors true 2024-06-15T10:31:45.790 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:45.790 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 4 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: osd.3 up in weight 1 up_from 117 up_thru 111 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6826/1106745740,v1:127.0.0.1:6827/1106745740] [v2:127.0.0.1:6828/1106745740,v1:127.0.0.1:6829/1106745740] exists,up 25307158-8c45-4982-8fa8-1f0fe831e9d5 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:31:45.791 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:31:45.792 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 1 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 2 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 3 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 4 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 5 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 6 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 7 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 8 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: 9' 2024-06-15T10:31:45.793 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:31:45.794 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:45.794 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:31:45.794 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836609 2024-06-15T10:31:45.794 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836609 2024-06-15T10:31:45.794 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609' 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:24007ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 4 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:47.805 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 5 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: osd.5 up in weight 1 up_from 120 up_thru 120 down_at 113 last_clean_interval [108,112) [v2:127.0.0.1:6842/47597818,v1:127.0.0.1:6843/47597818] [v2:127.0.0.1:6844/47597818,v1:127.0.0.1:6845/47597818] exists,up b9cfb8b2-0e6d-4a3d-bc91-50f54a8b2c3f 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:31:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:31:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 1 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 2 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 3 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 4 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 5 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 6 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 7 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 8 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: 9' 2024-06-15T10:31:47.808 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:31:48.098 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ce3: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673083 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673083 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083' 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509554 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509554 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554' 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.099 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634' 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182492 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182492 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492' 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-06-15T10:31:48.100 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492 5-515396075522' 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855437 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855437 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492 5-515396075522 6-150323855437' 2024-06-15T10:31:48.101 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.248 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:49.248 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691910 2024-06-15T10:31:49.248 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691910 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492 5-515396075522 6-150323855437 7-171798691910' 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528385 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528385 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492 5-515396075522 6-150323855437 7-171798691910 8-193273528385' 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664208 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664208 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836609 1-42949673083 2-64424509554 3-502511173634 4-107374182492 5-515396075522 6-150323855437 7-171798691910 8-193273528385 9-438086664208' 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:49.249 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836609 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836609 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836609 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836609' 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.0 seq 21474836609 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836609 -lt 21474836609 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673083 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673083 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673083 2024-06-15T10:31:49.251 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673083' 2024-06-15T10:31:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waitph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:31:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836610 2024-06-15T10:31:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836610 2024-06-15T10:31:49.554 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610' 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673084 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673084 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084' 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509556 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509556 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556' 2024-06-15T10:31:49.555 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173636 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173636 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636' 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182494 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182494 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494' 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.556 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494 5-515396075523' 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:31:49.557 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855438 2024-06-15T10:31:50.171 INFO:tasks.workunit.client.0.smithi116.stderr:ing osd.1 seq 42949673083 2024-06-15T10:31:50.171 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673084 -lt 42949673083 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509554 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509554 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509554 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509554' 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.2 seq 64424509554 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509555 -lt 64424509554 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.172 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-502511173634 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-502511173634 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 502511173634' 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.3 seq 502511173634 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173635 -lt 502511173634 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182492 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182492 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182492 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182492' 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.4 seq 107374182492 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182493 -lt 107374182492 2024-06-15T10:31:50.174 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.215 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cep240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855438 2024-06-15T10:31:50.215 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494 5-515396075523 6-150323855438' 2024-06-15T10:31:50.215 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691911 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691911 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494 5-515396075523 6-150323855438 7-171798691911' 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528386 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528386 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494 5-515396075523 6-150323855438 7-171798691911 8-193273528386' 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:31:50.216 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664209 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664209 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836610 1-42949673084 2-64424509556 3-502511173636 4-107374182494 5-515396075523 6-150323855438 7-171798691911 8-193273528386 9-438086664209' 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836610 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836610 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836610 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.0 seq 21474836610 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836610 -lt 21474836610 2024-06-15T10:31:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.218 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673084 2024-06-15T10:31:50.218 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.930 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalohtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075522 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075522 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.5 seq 515396075522 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075522 -lt 515396075522 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855437 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.931 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855437 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855437 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855437' 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.6 seq 150323855437 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855439 -lt 150323855437 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691910 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691910 2024-06-15T10:31:50.932 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691910 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.7 seq 171798691910 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691911 -lt 171798691910 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528385 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.933 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: ecne/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673084 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673084 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673084' 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.1 seq 42949673084 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673084 -lt 42949673084 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509556 2024-06-15T10:31:50.935 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509556 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509556 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.2 seq 64424509556 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509556 -lt 64424509556 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-502511173636 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:31:50.936 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-502511173636 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173636 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 502511173636' 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.3 seq 502511173636 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173636 -lt 502511173636 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182494 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182494 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182494 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.4 seq 107374182494 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182494 -lt 107374182494 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075523 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:31:52.052 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075523 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075523' 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.5 seq 515396075523 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075524 -lt 515396075523 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855438 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855438 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:52.053 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855438 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855438' 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.6 seq 150323855438 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855439 -lt 150323855438 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691911 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691911 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691911 2024-06-15T10:31:52.054 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691911' 2024-06-15T10:31:52.055 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.7 seq 171798691911 2024-06-15T10:31:52.055 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:31:52.055 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691911 -lt 171798691911 2024-06-15T10:31:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /hho 8-193273528385 2024-06-15T10:31:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528385 2024-06-15T10:31:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528385' 2024-06-15T10:31:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.8 seq 193273528385 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528386 -lt 193273528385 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664208 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664208 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664208 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664208' 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: waiting osd.9 seq 438086664208 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664209 -lt 438086664208 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:52.240 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:31:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /homeome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528386 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528386 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528386 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528386' 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.8 seq 193273528386 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528386 -lt 193273528386 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:31:53.079 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664209 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664209 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664209 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664209' 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: waiting osd.9 seq 438086664209 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664210 -lt 438086664209 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:53.080 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:53.081 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:53.310 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.clien/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:31:53.310 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:31:53.310 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:53.311 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:53.312 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:31:53.313 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:53.313 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:31:53.313 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clt.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:31:54.195 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:54.196 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:54.197 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:54.473 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standaloneone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:54.473 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:54.473 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:54.473 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:54.473 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:31:54.474 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:54.475 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:54.901 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:31:54.901 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:31:54.901 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:31:54.901 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:31:54.901 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:54.902 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:54.903 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:54.903 INFO:tasks.workunit.client.0.smithi116.stderr:240075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:54.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:54.988 INFO:tasks.workunit.client.0.smithi116.stderr:h:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:31:54.988 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:31:54.988 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:31:54.988 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:240073: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 240071 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:31:54.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:31:54.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:31:55.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:55.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:55.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:55.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:55.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:55.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:31:55.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:31:55.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:31:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:31:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:31:55.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:31:55.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:31:55.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:31:55.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:31:55.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:31:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:31:55.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:31:55.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:31:55.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:31:55.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:55.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:55.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:31:55.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:31:55.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:31:55.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:31:55.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:31:55.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:31:55.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:31:55.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:31:55.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:31:55.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:31:55.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:31:55.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:31:55.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:31:55.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:31:55.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:31:55.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:31:55.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:31:55.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:31:55.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:31:55.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:31:55.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:31:55.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:31:55.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:31:55.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:31:55.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:31:55.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:31:55.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:31:55.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:31:55.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:31:55.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:31:55.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:55.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:31:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:31:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:31:55.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:31:55.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:31:55.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:55.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:55.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:31:55.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:31:55.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:31:55.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:31:55.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:31:55.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-06-15T10:31:55.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:31:55.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:31:55.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:31:55.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:55.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:55.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:31:56.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:31:56.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:31:56.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:31:56.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:31:56.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:31:56.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:31:56.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:31:56.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:31:56.124 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:31:56.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:31:56.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:31:56.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:31:56.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:31:56.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:31:56.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:31:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:31:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:31:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:31:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:31:56.243 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:56.244 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:56.244 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:31:56.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:31:56.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:31:56.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:31:57.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:57.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:57.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:31:57.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:31:57.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-06-15T10:31:57.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:31:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:31:57.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:31:57.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:31:57.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:31:57.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5170d91-04a2-44e6-947c-ef2f750b1913 2024-06-15T10:31:57.297 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 a5170d91-04a2-44e6-947c-ef2f750b1913 2024-06-15T10:31:57.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a5170d91-04a2-44e6-947c-ef2f750b1913' 2024-06-15T10:31:57.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:31:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdbW1mn4zkEhAAz3dbjvQ50FlzWN+WncrPzg== 2024-06-15T10:31:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdbW1mn4zkEhAAz3dbjvQ50FlzWN+WncrPzg=="}' 2024-06-15T10:31:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5170d91-04a2-44e6-947c-ef2f750b1913 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:31:57.589 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:31:57.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:31:57.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdbW1mn4zkEhAAz3dbjvQ50FlzWN+WncrPzg== --osd-uuid a5170d91-04a2-44e6-947c-ef2f750b1913 2024-06-15T10:31:57.642 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.639+0000 7fbb36b937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:57.657 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.655+0000 7fbb36b937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:57.658 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.655+0000 7fbb36b937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:31:57.658 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.655+0000 7fbb36b937c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:31:57.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.655+0000 7fbb36b937c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:31:57.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.655+0000 7fbb36b937c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:31:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:31:57.755+0000 7fbb36b937c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:32:04.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:32:04.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:32:04.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:32:04.256 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:32:04.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:32:04.529 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:32:04.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:32:04.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:32:04.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:32:04.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:32:04.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:32:04.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:04.559+0000 7f810ea9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:04.598 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:04.595+0000 7f810ea9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:04.599 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:04.599+0000 7f810ea9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:04.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:32:04.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:32:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:32:04.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:04.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:05.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:05.157 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:05.155+0000 7f810ea9b7c0 -1 Falling back to public interface 2024-06-15T10:32:06.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:06.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:06.039 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:32:06.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:32:06.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:06.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:06.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:07.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:07.282 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:32:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:32:07.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:07.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:07.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:08.011 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:08.007+0000 7f810ea9b7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:32:08.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:08.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:08.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:32:08.525 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:32:08.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:08.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:08.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:09.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:09.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:09.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:32:09.876 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:32:09.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:09.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:10.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:11.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:11.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:11.119 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:32:11.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:32:11.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:11.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:11.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:12.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:12.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:12.365 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:32:12.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:32:12.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:12.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:12.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:13.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:13.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:13.608 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:32:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:32:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:13.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:13.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:14.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:14.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:14.913 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:32:14.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:32:14.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:14.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:15.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:16.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:16.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:16.158 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:32:16.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:32:16.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:16.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:16.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:17.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:17.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:17.402 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:32:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:32:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:17.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:18.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:18.771 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:32:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:32:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:18.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:19.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:20.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:20.013 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:32:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:32:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:20.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:20.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:21.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:21.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:21.256 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:32:21.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:32:21.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:21.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:21.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:22.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:22.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:22.663 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:32:22.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:32:22.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:22.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:22.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:23.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:23.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:23.991 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:32:23.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:32:23.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:23.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:24.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:25.774 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:32:25.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:32:25.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:25.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:26.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:27.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:27.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:27.441 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:32:27.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:32:27.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:27.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:29.258 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:32:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:32:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:29.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:29.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:30.850 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:32:30.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:32:30.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:30.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:32.603 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:32:32.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:32:32.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:32.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:32.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:33.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:33.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:33.873 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:32:33.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:32:33.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:33.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:32:34.161 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2484987736,v1:127.0.0.1:6803/2484987736] [v2:127.0.0.1:6804/2484987736,v1:127.0.0.1:6805/2484987736] exists,up a5170d91-04a2-44e6-947c-ef2f750b1913 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:32:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:32:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:32:34.166 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:32:34.166 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:32:34.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:32:34.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:32:34.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:32:34.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7cd835bd-fdd5-4502-93e2-4349947247c0 2024-06-15T10:32:34.169 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 7cd835bd-fdd5-4502-93e2-4349947247c0 2024-06-15T10:32:34.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7cd835bd-fdd5-4502-93e2-4349947247c0' 2024-06-15T10:32:34.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:32:34.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCbW1mHGFaCxAAkyC87zw8nwsHbgVXRKOcqw== 2024-06-15T10:32:34.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCbW1mHGFaCxAAkyC87zw8nwsHbgVXRKOcqw=="}' 2024-06-15T10:32:34.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7cd835bd-fdd5-4502-93e2-4349947247c0 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:32:34.516 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:32:34.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:32:34.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCbW1mHGFaCxAAkyC87zw8nwsHbgVXRKOcqw== --osd-uuid 7cd835bd-fdd5-4502-93e2-4349947247c0 2024-06-15T10:32:34.570 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.571+0000 7f57f2c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:34.601 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.603+0000 7f57f2c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:34.603 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.603+0000 7f57f2c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:34.603 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.607+0000 7f57f2c627c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:32:34.603 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.607+0000 7f57f2c627c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:32:34.603 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.607+0000 7f57f2c627c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:32:34.677 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:34.679+0000 7f57f2c627c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:32:41.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:32:41.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:32:41.470 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:32:41.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:32:41.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:32:41.739 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:32:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:32:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:32:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:32:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:32:41.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:32:41.773 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:41.775+0000 7f713098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:41.775 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:41.779+0000 7f713098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:41.776 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:41.779+0000 7f713098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:41.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:32:41.969 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:32:41.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:41.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:42.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:42.350 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:42.351+0000 7f713098e7c0 -1 Falling back to public interface 2024-06-15T10:32:43.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:43.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:43.195 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:32:43.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:32:43.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:43.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:43.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:44.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:44.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:44.431 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:32:44.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:32:44.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:44.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:44.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:45.108 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:32:45.111+0000 7f713098e7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:32:45.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:45.678 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:32:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:32:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:46.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:47.055 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:32:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:32:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:47.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:47.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:48.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:48.450 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:32:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:32:48.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:48.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:48.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:49.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:49.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:49.827 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:32:49.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:32:49.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:49.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:50.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:51.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:51.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:51.073 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:32:51.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:32:51.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:51.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:51.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:52.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:52.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:52.307 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:32:52.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:32:52.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:52.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:52.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:53.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:32:53.715 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:32:53.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:53.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:54.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:55.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:55.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:55.054 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:32:55.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:32:55.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:55.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:55.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:56.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:32:56.297 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:32:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:56.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:56.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:57.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:57.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:57.676 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:32:57.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:32:57.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:57.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:58.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:32:59.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:32:59.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:32:59.057 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:32:59.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:32:59.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:32:59.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:32:59.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:00.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:00.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:00.301 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:33:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:33:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:00.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:00.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:01.590 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:33:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:33:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:02.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:03.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:03.167 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:33:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:33:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:03.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:03.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:04.888 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:33:04.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:33:04.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:04.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:05.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:06.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:06.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:06.467 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:33:06.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:33:06.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:06.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:07.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:08.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:08.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:08.049 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:33:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:33:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:08.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:08.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:09.451 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:33:09.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:33:09.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:09.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:09.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:10.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:10.989 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:33:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:33:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:11.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:12.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:12.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:12.374 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:33:12.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:33:12.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:12.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:33:12.630 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2142042606,v1:127.0.0.1:6811/2142042606] [v2:127.0.0.1:6812/2142042606,v1:127.0.0.1:6813/2142042606] exists,up 7cd835bd-fdd5-4502-93e2-4349947247c0 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:33:12.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:33:12.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:33:12.635 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:33:12.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:33:12.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:33:12.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:33:12.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:33:12.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=452860d6-b6cf-4e61-aa6d-05c16e369d08 2024-06-15T10:33:12.638 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 452860d6-b6cf-4e61-aa6d-05c16e369d08 2024-06-15T10:33:12.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 452860d6-b6cf-4e61-aa6d-05c16e369d08' 2024-06-15T10:33:12.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:33:12.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDobW1mJzZFJxAA+HPpZJfFgga1yJ6eMUXafw== 2024-06-15T10:33:12.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDobW1mJzZFJxAA+HPpZJfFgga1yJ6eMUXafw=="}' 2024-06-15T10:33:12.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 452860d6-b6cf-4e61-aa6d-05c16e369d08 -i td/osd-scrub-repair/2/new.json 2024-06-15T10:33:12.966 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:33:12.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:33:12.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDobW1mJzZFJxAA+HPpZJfFgga1yJ6eMUXafw== --osd-uuid 452860d6-b6cf-4e61-aa6d-05c16e369d08 2024-06-15T10:33:13.014 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.014+0000 7f3a949367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:13.076 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.078+0000 7f3a949367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:13.078 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.078+0000 7f3a949367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:13.078 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.078+0000 7f3a949367c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:33:13.079 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.078+0000 7f3a949367c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:33:13.079 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.078+0000 7f3a949367c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:33:13.154 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:13.154+0000 7f3a949367c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:33:20.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:33:20.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:33:20.246 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:33:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:33:20.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:33:20.544 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:33:20.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:33:20.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:33:20.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:33:20.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:33:20.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:33:20.577 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:20.578+0000 7fe82bd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:20.579 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:20.582+0000 7fe82bd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:20.581 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:20.582+0000 7fe82bd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:20.784 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:33:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:33:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:20.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:21.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:21.146 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:21.146+0000 7fe82bd127c0 -1 Falling back to public interface 2024-06-15T10:33:22.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:22.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:22.044 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:33:22.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:33:22.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:22.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:22.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:23.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:23.278 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:33:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:33:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:23.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:23.879 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:23.882+0000 7fe82bd127c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:33:24.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:24.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:24.556 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:33:24.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:33:24.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:24.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:24.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:25.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:25.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:25.817 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:33:25.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:33:25.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:25.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:26.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:27.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:27.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:27.135 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:33:27.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:33:27.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:27.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:27.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:28.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:28.376 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:33:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:33:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:28.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:28.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:29.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:29.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:29.704 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:33:29.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:33:29.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:29.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:29.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:30.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:30.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:30.991 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:33:30.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:33:30.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:30.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:31.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:32.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:32.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:32.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:33:32.231 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:33:32.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:32.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:32.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:33.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:33.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:33.476 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:33:33.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:33:33.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:33.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:33.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:34.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:34.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:34.811 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:33:34.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:33:34.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:34.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:35.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:36.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:36.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:36.116 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:33:36.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:33:36.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:36.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:36.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:37.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:37.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:37.358 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:33:37.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:33:37.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:37.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:37.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:38.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:38.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:38.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:33:38.620 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:33:38.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:38.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:39.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:40.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:40.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:40.205 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:33:40.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:33:40.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:40.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:40.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:41.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:41.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:41.861 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:33:41.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:33:41.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:41.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:42.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:43.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:43.183 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:33:43.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:33:43.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:43.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:43.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:44.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:44.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:44.875 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:33:44.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:33:44.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:44.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:45.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:46.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:46.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:46.381 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:33:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:33:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:46.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:46.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:47.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:47.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:47.634 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:33:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:33:47.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:47.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:48.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:49.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:49.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:49.202 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:33:49.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:33:49.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:49.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:49.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:50.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:33:50.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:50.719 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:33:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:33:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:50.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:33:50.957 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1706359232,v1:127.0.0.1:6819/1706359232] [v2:127.0.0.1:6820/1706359232,v1:127.0.0.1:6821/1706359232] exists,up 452860d6-b6cf-4e61-aa6d-05c16e369d08 2024-06-15T10:33:50.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:33:50.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:33:50.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:33:50.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:33:50.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:33:50.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:33:50.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a51f6c65-8c89-467b-b441-fcc49e4149e0' 2024-06-15T10:33:50.964 INFO:tasks.workunit.client.0.smithi116.stdout:add osd3 a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:33:50.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:33:50.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAObm1mWqvlOhAARI5PCRbnOaI1rZwNdwC4rA== 2024-06-15T10:33:50.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAObm1mWqvlOhAARI5PCRbnOaI1rZwNdwC4rA=="}' 2024-06-15T10:33:50.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a51f6c65-8c89-467b-b441-fcc49e4149e0 -i td/osd-scrub-repair/3/new.json 2024-06-15T10:33:51.330 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:33:51.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-06-15T10:33:51.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAObm1mWqvlOhAARI5PCRbnOaI1rZwNdwC4rA== --osd-uuid a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:33:51.388 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.390+0000 7fc6d8e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:51.440 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.442+0000 7fc6d8e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:51.442 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.442+0000 7fc6d8e1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:51.443 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.442+0000 7fc6d8e1d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:33:51.443 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.442+0000 7fc6d8e1d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:33:51.443 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.442+0000 7fc6d8e1d7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-06-15T10:33:51.548 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:51.550+0000 7fc6d8e1d7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-06-15T10:33:58.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-06-15T10:33:58.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:33:58.365 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd3 key to auth repository 2024-06-15T10:33:58.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-06-15T10:33:58.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:33:58.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-06-15T10:33:58.638 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.3 2024-06-15T10:33:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:33:58.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:33:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:33:58.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:33:58.665 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:58.665+0000 7f97e75d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:58.667 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:58.669+0000 7f97e75d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:58.669 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:58.669+0000 7f97e75d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:33:58.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:33:58.875 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:33:58.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:33:58.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:33:58.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:33:59.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:33:59.206 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:33:59.205+0000 7f97e75d97c0 -1 Falling back to public interface 2024-06-15T10:34:00.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:00.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:00.112 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:34:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:34:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:00.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:00.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:01.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:01.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:01.344 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:34:01.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:34:01.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:01.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:01.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:01.969 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:01.969+0000 7f97e75d97c0 -1 osd.3 0 log_to_monitors true 2024-06-15T10:34:02.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:02.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:02.588 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:34:02.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:34:02.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:02.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:02.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:03.962 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:34:03.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:34:03.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:03.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:04.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:05.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:05.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:05.263 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:34:05.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:34:05.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:05.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:05.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:06.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:06.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:06.509 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:34:06.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:34:06.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:06.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:07.904 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:34:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:34:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:08.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:09.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:09.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:09.147 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:34:09.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:34:09.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:09.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:09.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:10.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:10.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:10.388 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:34:10.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:34:10.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:10.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:10.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:11.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:11.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:11.690 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:34:11.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:34:11.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:11.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:12.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:13.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:13.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:13.008 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:34:13.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:34:13.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:13.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:14.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:14.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:14.249 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:34:14.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:34:14.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:14.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:14.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:15.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:15.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:15.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:34:15.684 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:34:15.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:15.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:15.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:16.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:16.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:16.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:34:16.997 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:34:16.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:16.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:17.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:18.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:18.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:18.474 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:34:18.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:34:18.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:18.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:19.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:20.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:20.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:20.061 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:34:20.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:34:20.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:20.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:20.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:21.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:21.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:21.578 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:34:21.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:34:21.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:21.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:22.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:23.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:23.217 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:34:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:34:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:23.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:23.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:24.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:24.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:24.820 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:34:24.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:34:24.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:24.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:25.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:26.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:26.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:26.210 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:34:26.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:34:26.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:26.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:26.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:27.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:27.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:27.459 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:34:27.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:34:27.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:27.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:28.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:29.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:29.034 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:34:29.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:34:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:34:29.348 INFO:tasks.workunit.client.0.smithi116.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3753712588,v1:127.0.0.1:6827/3753712588] [v2:127.0.0.1:6828/3753712588,v1:127.0.0.1:6829/3753712588] exists,up a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:34:29.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:34:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:34:29.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4 2024-06-15T10:34:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4' 2024-06-15T10:34:29.356 INFO:tasks.workunit.client.0.smithi116.stdout:add osd4 8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4 2024-06-15T10:34:29.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:34:29.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1bm1ml8+eFhAAC9Hi7+k6D3fsST7oN4yvFA== 2024-06-15T10:34:29.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1bm1ml8+eFhAAC9Hi7+k6D3fsST7oN4yvFA=="}' 2024-06-15T10:34:29.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4 -i td/osd-scrub-repair/4/new.json 2024-06-15T10:34:29.689 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:34:29.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-06-15T10:34:29.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1bm1ml8+eFhAAC9Hi7+k6D3fsST7oN4yvFA== --osd-uuid 8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4 2024-06-15T10:34:29.744 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.745+0000 7f643eb9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:29.783 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.785+0000 7f643eb9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:29.784 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.785+0000 7f643eb9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:29.785 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.785+0000 7f643eb9e7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:34:29.785 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.785+0000 7f643eb9e7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:34:29.785 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.785+0000 7f643eb9e7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-06-15T10:34:29.865 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:29.865+0000 7f643eb9e7c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-06-15T10:34:36.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-06-15T10:34:36.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:34:36.833 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd4 key to auth repository 2024-06-15T10:34:36.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-06-15T10:34:36.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:34:37.103 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.4 2024-06-15T10:34:37.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-06-15T10:34:37.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:34:37.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:34:37.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:34:37.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:34:37.133 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:37.133+0000 7f5cd6f4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:37.136 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:37.137+0000 7f5cd6f4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:37.137 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:37.137+0000 7f5cd6f4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:34:37.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:37.371 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:34:37.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:34:37.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:37.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:37.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:37.714 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:37.713+0000 7f5cd6f4b7c0 -1 Falling back to public interface 2024-06-15T10:34:38.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:38.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:38.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:34:38.637 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:34:38.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:38.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:38.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:39.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:39.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:39.868 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:34:39.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:34:39.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:39.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:40.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:40.489 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:34:40.489+0000 7f5cd6f4b7c0 -1 osd.4 0 log_to_monitors true 2024-06-15T10:34:41.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:41.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:41.112 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:34:41.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:34:41.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:41.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:41.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:42.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:42.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:42.542 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:34:42.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:34:42.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:42.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:42.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:43.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:43.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:43.960 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:34:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:34:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:43.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:44.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:45.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:45.201 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:34:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:34:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:45.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:45.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:46.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:46.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:46.541 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:34:46.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:34:46.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:46.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:46.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:47.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:47.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:47.860 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:34:47.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:34:47.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:47.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:48.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:49.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:49.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:49.106 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:34:49.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:34:49.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:49.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:49.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:50.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:50.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:50.541 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:34:50.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:34:50.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:50.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:50.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:51.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:51.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:51.880 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:34:51.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:34:51.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:51.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:53.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:53.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:53.123 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:34:53.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:34:53.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:53.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:53.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:54.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:54.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:54.531 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:34:54.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:34:54.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:54.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:54.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:55.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:55.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:55.910 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:34:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:34:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:56.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:57.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:57.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:57.154 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:34:57.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:34:57.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:57.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:57.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:34:58.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:34:58.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:34:58.744 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:34:58.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:34:58.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:34:58.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:34:59.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:00.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:00.422 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:35:00.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:35:00.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:00.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:00.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:01.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:01.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:01.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:35:01.884 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:35:01.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:01.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:02.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:03.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:03.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:03.814 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:35:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:35:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:04.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:05.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:05.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:05.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:35:05.126 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:35:05.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:05.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:05.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:06.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:06.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:06.714 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:35:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:35:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:06.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:07.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:08.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:08.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:08.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:35:08.080 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:35:08.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:08.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-06-15T10:35:08.322 INFO:tasks.workunit.client.0.smithi116.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4005377133,v1:127.0.0.1:6835/4005377133] [v2:127.0.0.1:6836/4005377133,v1:127.0.0.1:6837/4005377133] exists,up 8e4c6e24-ed87-4c35-9dbe-5f8ed72deed4 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-06-15T10:35:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:35:08.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:35:08.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:35:08.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:35:08.326 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:35:08.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:35:08.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:35:08.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:35:08.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:35:08.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:35:08.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:35:08.328 INFO:tasks.workunit.client.0.smithi116.stdout:add osd5 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:35:08.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 907a0d98-b3d5-4486-b9bb-78712b04c347' 2024-06-15T10:35:08.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:35:08.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcbm1m/OYSFRAA9PKOgzaCJ5VEhbXkkJPkbA== 2024-06-15T10:35:08.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcbm1m/OYSFRAA9PKOgzaCJ5VEhbXkkJPkbA=="}' 2024-06-15T10:35:08.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 907a0d98-b3d5-4486-b9bb-78712b04c347 -i td/osd-scrub-repair/5/new.json 2024-06-15T10:35:08.742 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:35:09.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcbm1m/OYSFRAA9PKOgzaCJ5VEhbXkkJPkbA== --osd-uuid 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.796+0000 7f12b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.812+0000 7f12b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.812+0000 7f12b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.812+0000 7f12b19287c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.812+0000 7f12b19287c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.812+0000 7f12b19287c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-06-15T10:35:09.009 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:08.928+0000 7f12b19287c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-06-15T10:35:15.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-06-15T10:35:15.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:35:15.939 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd5 key to auth repository 2024-06-15T10:35:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-06-15T10:35:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:35:16.213 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.5 2024-06-15T10:35:16.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-06-15T10:35:16.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:35:16.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:35:16.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:35:16.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:35:16.246 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:16.244+0000 7fb4b0d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:16.284 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:16.284+0000 7fb4b0d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:16.285 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:16.284+0000 7fb4b0d247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:16.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:35:16.486 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:35:16.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:16.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:16.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:16.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:16.836+0000 7fb4b0d247c0 -1 Falling back to public interface 2024-06-15T10:35:17.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:17.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:17.739 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:35:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:35:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:17.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:18.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:18.981 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:35:18.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:35:18.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:18.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:19.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:19.563 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:19.560+0000 7fb4b0d247c0 -1 osd.5 0 log_to_monitors true 2024-06-15T10:35:20.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:20.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:20.221 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:35:20.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:35:20.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:20.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:20.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:21.584 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:35:21.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:35:21.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:21.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:21.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:22.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:22.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:22.884 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:35:22.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:35:22.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:22.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:24.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:24.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:24.203 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:35:24.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:35:24.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:24.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:24.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:25.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:25.574 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:35:25.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:25.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:35:25.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:25.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:25.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:26.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:26.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:26.817 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:35:26.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:35:26.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:26.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:27.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:28.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:28.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:28.143 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:35:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:35:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:28.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:28.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:29.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:29.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:29.627 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:35:29.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:35:29.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:29.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:29.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:30.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:30.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:30.873 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:35:30.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:35:30.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:30.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:32.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:32.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:32.112 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:35:32.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:35:32.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:32.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:32.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:33.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:33.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:33.371 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:35:33.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:35:33.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:33.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:34.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:34.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:34.736 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:35:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:35:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:34.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:35.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:35.976 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:35:35.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:35:35.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:35.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:36.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:37.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:37.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:37.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:35:37.281 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:35:37.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:37.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:37.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:38.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:38.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:38.890 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:35:38.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:35:38.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:38.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:39.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:40.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:40.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:40.356 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:35:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:35:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:40.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:40.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:41.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:41.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:41.896 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:35:41.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:35:41.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:41.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:42.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:43.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:43.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:43.722 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:35:43.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:35:43.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:43.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:43.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:44.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:44.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:44.973 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:35:44.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:35:44.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:44.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:45.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:46.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:46.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:46.669 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:35:46.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:35:46.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:46.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:46.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:47.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:47.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:47.914 INFO:tasks.workunit.client.0.smithi116.stdout:23 2024-06-15T10:35:47.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-06-15T10:35:47.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:47.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:35:48.153 INFO:tasks.workunit.client.0.smithi116.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3848532857,v1:127.0.0.1:6843/3848532857] [v2:127.0.0.1:6844/3848532857,v1:127.0.0.1:6845/3848532857] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-06-15T10:35:48.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:35:48.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:35:48.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:35:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1 2024-06-15T10:35:48.161 INFO:tasks.workunit.client.0.smithi116.stdout:add osd6 ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1 2024-06-15T10:35:48.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1' 2024-06-15T10:35:48.161 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:35:48.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEbm1mESMBCxAAlFODlp+b2HoV/E43xNS4uA== 2024-06-15T10:35:48.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEbm1mESMBCxAAlFODlp+b2HoV/E43xNS4uA=="}' 2024-06-15T10:35:48.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1 -i td/osd-scrub-repair/6/new.json 2024-06-15T10:35:48.637 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:35:48.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-06-15T10:35:48.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEbm1mESMBCxAAlFODlp+b2HoV/E43xNS4uA== --osd-uuid ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1 2024-06-15T10:35:48.692 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.692+0000 7fdf023ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:48.756 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.756+0000 7fdf023ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:48.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.756+0000 7fdf023ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:48.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.756+0000 7fdf023ef7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:35:48.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.756+0000 7fdf023ef7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:35:48.759 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.756+0000 7fdf023ef7c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-06-15T10:35:48.840 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:48.840+0000 7fdf023ef7c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-06-15T10:35:55.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-06-15T10:35:55.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:35:55.898 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd6 key to auth repository 2024-06-15T10:35:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-06-15T10:35:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:35:56.171 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.6 2024-06-15T10:35:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-06-15T10:35:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:35:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:35:56.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:35:56.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:35:56.203 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:56.204+0000 7fd61beb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:56.205 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:56.204+0000 7fd61beb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:56.207 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:56.208+0000 7fd61beb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:35:56.440 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:35:56.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:56.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:35:56.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:56.780 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:56.780+0000 7fd61beb97c0 -1 Falling back to public interface 2024-06-15T10:35:57.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:57.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:57.721 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:35:57.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:35:57.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:57.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:35:57.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:58.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:35:58.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:35:58.963 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:35:58.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:35:58.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:35:58.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:35:59.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:35:59.446 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:35:59.444+0000 7fd61beb97c0 -1 osd.6 0 log_to_monitors true 2024-06-15T10:36:00.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:00.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:00.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:36:00.204 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:36:00.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:00.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:00.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:01.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:01.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:01.499 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:36:01.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:36:01.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:01.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:01.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:02.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:02.759 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:36:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:36:02.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:02.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:03.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:04.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:04.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:04.005 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:36:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:36:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:04.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:04.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:05.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:05.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:05.415 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:36:05.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:36:05.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:05.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:05.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:06.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:06.659 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:36:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:36:06.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:06.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:06.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:07.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:07.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:07.903 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:36:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:36:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:08.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:09.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:09.175 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:36:09.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:36:09.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:09.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:09.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:10.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:10.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:10.534 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:36:10.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:36:10.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:10.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:10.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:11.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:11.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:11.900 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:36:11.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:36:11.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:11.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:12.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:13.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:13.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:13.151 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:36:13.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:36:13.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:13.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:13.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:14.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:14.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:14.532 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:36:14.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:36:14.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:14.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:15.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:16.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:16.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:16.172 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:36:16.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:36:16.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:16.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:16.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:17.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:17.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:17.675 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:36:17.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:36:17.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:17.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:17.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:18.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:18.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:18.917 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:36:18.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:36:18.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:18.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:19.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:20.507 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:36:20.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:36:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:20.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:21.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:22.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:22.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:22.127 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:36:22.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:36:22.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:22.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:22.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:23.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:23.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:23.566 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:36:23.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:36:23.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:23.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:24.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:25.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:25.363 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:36:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:36:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:26.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:27.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:27.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:27.017 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:36:27.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:36:27.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:27.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-06-15T10:36:27.258 INFO:tasks.workunit.client.0.smithi116.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3774762152,v1:127.0.0.1:6851/3774762152] [v2:127.0.0.1:6852/3774762152,v1:127.0.0.1:6853/3774762152] exists,up ed5cc0f3-a933-4241-8ad7-e909b6e5ebd1 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-06-15T10:36:27.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:36:27.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:36:27.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:36:27.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-06-15T10:36:27.265 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:36:27.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=defa7c3f-ddd2-436d-89df-ff891ec604d5 2024-06-15T10:36:27.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 defa7c3f-ddd2-436d-89df-ff891ec604d5' 2024-06-15T10:36:27.265 INFO:tasks.workunit.client.0.smithi116.stdout:add osd7 defa7c3f-ddd2-436d-89df-ff891ec604d5 2024-06-15T10:36:27.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:36:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrbm1mDZABERAAL3rq8Q0OEGFxWUz8reY5LA== 2024-06-15T10:36:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrbm1mDZABERAAL3rq8Q0OEGFxWUz8reY5LA=="}' 2024-06-15T10:36:27.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new defa7c3f-ddd2-436d-89df-ff891ec604d5 -i td/osd-scrub-repair/7/new.json 2024-06-15T10:36:27.593 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:36:27.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-06-15T10:36:27.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrbm1mDZABERAAL3rq8Q0OEGFxWUz8reY5LA== --osd-uuid defa7c3f-ddd2-436d-89df-ff891ec604d5 2024-06-15T10:36:27.648 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.647+0000 7fb3126b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:27.712 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.711+0000 7fb3126b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.711+0000 7fb3126b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.711+0000 7fb3126b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:36:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.711+0000 7fb3126b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:36:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.711+0000 7fb3126b27c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-06-15T10:36:27.835 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:27.835+0000 7fb3126b27c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-06-15T10:36:34.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-06-15T10:36:34.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:36:34.456 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd7 key to auth repository 2024-06-15T10:36:34.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-06-15T10:36:34.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:36:34.732 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.7 2024-06-15T10:36:34.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-06-15T10:36:34.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:36:34.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:36:34.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:36:34.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:36:34.764 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:34.763+0000 7f62cc29d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:34.767 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:34.767+0000 7f62cc29d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:34.768 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:34.767+0000 7f62cc29d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:36:34.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-06-15T10:36:34.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:36:34.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-06-15T10:36:34.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:34.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:35.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:35.349 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:35.347+0000 7f62cc29d7c0 -1 Falling back to public interface 2024-06-15T10:36:36.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:36.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:36.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:36:36.258 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:36:36.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:36.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:36.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:37.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:37.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:37.522 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:36:37.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:36:37.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:37.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:37.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:38.039 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:36:38.039+0000 7f62cc29d7c0 -1 osd.7 0 log_to_monitors true 2024-06-15T10:36:38.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:38.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:38.762 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:36:38.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:36:38.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:38.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:39.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:40.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:40.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:40.187 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:36:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:36:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:40.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:40.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:41.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:41.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:41.496 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:36:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:36:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:41.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:42.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:42.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:42.843 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:36:42.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:36:42.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:42.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:43.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:44.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:44.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:44.167 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:36:44.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:36:44.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:44.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:44.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:45.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:45.509 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:36:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:36:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:45.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:45.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:46.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:46.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:46.751 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:36:46.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:36:46.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:47.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:48.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:48.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:48.033 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:36:48.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:36:48.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:48.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:48.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:49.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:49.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:49.278 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:36:49.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:36:49.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:49.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:49.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:50.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:50.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:50.521 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:36:50.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:36:50.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:50.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:50.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:51.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:51.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:51.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:36:51.776 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:36:51.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:51.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:52.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:53.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:53.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:53.117 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:36:53.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:36:53.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:53.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:53.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:54.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:54.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:54.359 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:36:54.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:36:54.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:54.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:54.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:55.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:55.969 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:36:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:36:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:56.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:57.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:57.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:57.710 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:36:57.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:36:57.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:57.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:58.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:36:59.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:36:59.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:36:59.116 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:36:59.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:36:59.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:36:59.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:36:59.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:00.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:00.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:00.656 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:37:00.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:37:00.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:00.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:37:01.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:02.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:02.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:02.155 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:37:02.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:37:02.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:02.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:37:02.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:03.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-06-15T10:37:03.399 INFO:tasks.workunit.client.0.smithi116.stdout:21 2024-06-15T10:37:03.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:03.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:37:03.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:04.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:04.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:04.644 INFO:tasks.workunit.client.0.smithi116.stdout:22 2024-06-15T10:37:04.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-06-15T10:37:04.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:04.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-06-15T10:37:04.880 INFO:tasks.workunit.client.0.smithi116.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2387067632,v1:127.0.0.1:6859/2387067632] [v2:127.0.0.1:6860/2387067632,v1:127.0.0.1:6861/2387067632] exists,up defa7c3f-ddd2-436d-89df-ff891ec604d5 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-06-15T10:37:04.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:37:04.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:37:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:37:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:37:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:37:04.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:37:04.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:37:04.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-06-15T10:37:04.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:37:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a14383a-316d-4eea-96f5-7286652ac940 2024-06-15T10:37:04.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 7a14383a-316d-4eea-96f5-7286652ac940' 2024-06-15T10:37:04.887 INFO:tasks.workunit.client.0.smithi116.stdout:add osd8 7a14383a-316d-4eea-96f5-7286652ac940 2024-06-15T10:37:04.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:37:04.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQbm1maBISNhAAuUf0wgBHYu1tZPYUJ1ACXg== 2024-06-15T10:37:04.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQbm1maBISNhAAuUf0wgBHYu1tZPYUJ1ACXg=="}' 2024-06-15T10:37:04.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a14383a-316d-4eea-96f5-7286652ac940 -i td/osd-scrub-repair/8/new.json 2024-06-15T10:37:05.221 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:37:05.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-06-15T10:37:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQbm1maBISNhAAuUf0wgBHYu1tZPYUJ1ACXg== --osd-uuid 7a14383a-316d-4eea-96f5-7286652ac940 2024-06-15T10:37:05.274 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.275+0000 7fc4add677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:05.289 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.291+0000 7fc4add677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:05.289 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.291+0000 7fc4add677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:05.290 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.291+0000 7fc4add677c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:37:05.290 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.291+0000 7fc4add677c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:37:05.290 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.291+0000 7fc4add677c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-06-15T10:37:05.422 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:05.423+0000 7fc4add677c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-06-15T10:37:11.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-06-15T10:37:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:37:11.781 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd8 key to auth repository 2024-06-15T10:37:11.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-06-15T10:37:11.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:37:12.057 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.8 2024-06-15T10:37:12.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-06-15T10:37:12.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:37:12.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:37:12.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:37:12.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:37:12.089 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:12.091+0000 7f7e9312c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:12.095+0000 7f7e9312c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:12.093 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:12.095+0000 7f7e9312c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:37:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:12.327 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:37:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:37:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:12.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:12.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:12.680 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:12.683+0000 7f7e9312c7c0 -1 Falling back to public interface 2024-06-15T10:37:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:13.604 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:37:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:37:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:13.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:14.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:14.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:14.854 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:37:14.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:37:14.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:14.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:15.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:15.199 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:15.202+0000 7f7e9312c7c0 -1 osd.8 0 log_to_monitors true 2024-06-15T10:37:16.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:16.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:16.099 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:37:16.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:37:16.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:16.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:16.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:17.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:17.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:17.502 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:37:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:37:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:17.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:18.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:18.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:18.752 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:37:18.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:37:18.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:18.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:19.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:20.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:20.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:20.089 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:37:20.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:37:20.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:20.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:20.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:21.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:21.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:21.330 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:37:21.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:37:21.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:21.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:21.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:22.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:37:22.577 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:37:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:22.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:22.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:23.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:23.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:23.873 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:37:23.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:37:23.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:23.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:24.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:25.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:25.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:25.201 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:37:25.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:37:25.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:25.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:25.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:26.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:26.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:26.446 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:37:26.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:37:26.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:26.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:26.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:27.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:27.726 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:37:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:37:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:27.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:28.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:28.972 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:37:28.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:37:28.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:28.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:29.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:30.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:30.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:30.449 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:37:30.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:37:30.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:30.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:30.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:31.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:31.701 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:37:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:37:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:31.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:32.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:33.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:33.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:33.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:37:33.537 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:37:33.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:33.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:34.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:35.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:35.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:35.359 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:37:35.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:37:35.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:35.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:35.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:36.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:36.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:36.746 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:37:36.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:37:36.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:36.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:37.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:38.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:38.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:38.508 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:37:38.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:37:38.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:38.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:38.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:39.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:39.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:39.750 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:37:39.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:37:39.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:39.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-06-15T10:37:39.991 INFO:tasks.workunit.client.0.smithi116.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/625245700,v1:127.0.0.1:6867/625245700] [v2:127.0.0.1:6868/625245700,v1:127.0.0.1:6869/625245700] exists,up 7a14383a-316d-4eea-96f5-7286652ac940 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-06-15T10:37:39.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:37:39.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:37:39.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-06-15T10:37:39.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:37:39.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:37:39.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:37:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:37:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:37:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:37:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:37:39.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:37:39.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:37:39.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:37:39.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:37:39.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:37:39.998 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:37:40.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:37:40.001 INFO:tasks.workunit.client.0.smithi116.stdout:add osd9 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:37:40.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3' 2024-06-15T10:37:40.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:37:40.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0bm1mKcEmARAA0+XYAkjZehtsdclPcNpNeA== 2024-06-15T10:37:40.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0bm1mKcEmARAA0+XYAkjZehtsdclPcNpNeA=="}' 2024-06-15T10:37:40.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 -i td/osd-scrub-repair/9/new.json 2024-06-15T10:37:40.299 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:37:40.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-06-15T10:37:40.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0bm1mKcEmARAA0+XYAkjZehtsdclPcNpNeA== --osd-uuid 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:37:40.353 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.354+0000 7fd0676317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:40.400 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.402+0000 7fd0676317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:40.402 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.402+0000 7fd0676317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:40.402 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.402+0000 7fd0676317c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:37:40.403 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.402+0000 7fd0676317c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:37:40.403 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.402+0000 7fd0676317c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-06-15T10:37:40.463 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:40.462+0000 7fd0676317c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-06-15T10:37:46.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-06-15T10:37:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:37:46.813 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd9 key to auth repository 2024-06-15T10:37:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-06-15T10:37:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:37:47.089 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.9 2024-06-15T10:37:47.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-06-15T10:37:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:37:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:37:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:37:47.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:37:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:47.122+0000 7fac4e25b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:47.155 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:47.158+0000 7fac4e25b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:47.157 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:47.158+0000 7fac4e25b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:37:47.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:47.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:37:47.347 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:37:47.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:47.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:47.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:47.714 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:47.714+0000 7fac4e25b7c0 -1 Falling back to public interface 2024-06-15T10:37:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:48.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:48.610 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:37:48.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:37:48.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:48.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:48.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:49.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:49.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:49.853 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:37:49.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:37:49.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:49.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:50.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:50.287 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:37:50.290+0000 7fac4e25b7c0 -1 osd.9 0 log_to_monitors true 2024-06-15T10:37:51.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:51.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:51.093 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:37:51.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:37:51.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:51.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:51.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:37:52.408 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:37:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:52.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:53.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:53.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:53.740 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:37:53.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:37:53.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:53.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:54.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:55.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:55.042 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:37:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:37:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:55.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:55.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:56.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:56.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:56.344 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:37:56.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:37:56.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:56.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:56.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:57.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:57.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:57.685 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:37:57.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:37:57.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:57.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:57.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:37:58.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:37:58.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:37:58.932 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:37:58.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:37:58.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:37:58.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:37:59.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:00.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:00.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:00.242 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:38:00.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:38:00.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:00.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:01.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:01.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:01.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:38:01.493 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:38:01.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:01.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:01.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:02.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:02.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:02.742 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:38:02.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:38:02.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:02.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:03.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:04.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:04.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:04.220 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:38:04.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:38:04.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:04.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:04.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:05.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:05.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:05.483 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:38:05.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:38:05.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:05.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:05.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:06.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:06.754 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:38:06.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:06.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:38:06.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:06.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:07.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:08.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:08.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:08.404 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:38:08.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:38:08.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:08.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:08.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:09.886 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:38:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:38:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:10.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:11.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:11.583 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:38:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:38:11.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:11.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:12.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:13.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:13.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:13.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:38:13.049 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:38:13.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:13.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:14.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:14.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:14.360 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T10:38:14.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T10:38:14.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:14.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:14.729 INFO:tasks.workunit.client.0.smithi116.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3484829332,v1:127.0.0.1:6875/3484829332] [v2:127.0.0.1:6876/3484829332,v1:127.0.0.1:6877/3484829332] exists,up 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:38:14.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:38:14.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:38:14.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:38:14.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-06-15T10:38:14.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:38:14.995 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:38:15.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:38:15.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:38:15.412 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:38:15.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:38:16.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:38:19.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-06-15T10:38:19.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:38:19.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:38:19.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:38:19.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:38:19.780 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:38:19.780 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:38:19.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:38:19.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:38:19.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:38:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:38:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:38:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:38:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:38:19.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:38:19.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:38:20.289 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:38:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:38:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:38:20.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:20.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:38:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-06-15T10:38:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-06-15T10:38:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-06-15T10:38:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:20.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:38:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-06-15T10:38:20.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-06-15T10:38:20.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025' 2024-06-15T10:38:20.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:20.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:38:20.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-06-15T10:38:20.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-06-15T10:38:20.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501' 2024-06-15T10:38:20.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:20.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:38:20.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345976 2024-06-15T10:38:20.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345976 2024-06-15T10:38:20.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976' 2024-06-15T10:38:20.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:20.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:38:21.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-06-15T10:38:21.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-06-15T10:38:21.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448' 2024-06-15T10:38:21.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:21.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018913 2024-06-15T10:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018913 2024-06-15T10:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448 5-128849018913' 2024-06-15T10:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:21.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:38:21.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-06-15T10:38:21.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-06-15T10:38:21.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448 5-128849018913 6-150323855387' 2024-06-15T10:38:21.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:21.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:38:21.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691860 2024-06-15T10:38:21.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691860 2024-06-15T10:38:21.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448 5-128849018913 6-150323855387 7-171798691860' 2024-06-15T10:38:21.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:21.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:38:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528337 2024-06-15T10:38:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528337 2024-06-15T10:38:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448 5-128849018913 6-150323855387 7-171798691860 8-193273528337' 2024-06-15T10:38:21.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:21.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:38:21.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-06-15T10:38:21.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-06-15T10:38:21.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673025 2-64424509501 3-85899345976 4-107374182448 5-128849018913 6-150323855387 7-171798691860 8-193273528337 9-210453397509' 2024-06-15T10:38:21.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:21.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-06-15T10:38:21.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:21.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:38:21.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-06-15T10:38:21.727 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:21.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-06-15T10:38:21.728 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836556 2024-06-15T10:38:21.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-06-15T10:38:21.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:38:21.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-06-15T10:38:21.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:21.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-06-15T10:38:21.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:21.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:38:21.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-06-15T10:38:21.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:21.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-06-15T10:38:21.963 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673025 2024-06-15T10:38:21.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-06-15T10:38:21.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:38:22.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-06-15T10:38:22.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:22.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:22.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-06-15T10:38:22.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:38:22.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-06-15T10:38:22.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:22.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-06-15T10:38:22.251 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509501 2024-06-15T10:38:22.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-06-15T10:38:22.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:38:22.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509501 -lt 64424509501 2024-06-15T10:38:22.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:22.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345976 2024-06-15T10:38:22.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:22.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:38:22.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345976 2024-06-15T10:38:22.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:22.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345976 2024-06-15T10:38:22.507 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345976 2024-06-15T10:38:22.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345976' 2024-06-15T10:38:22.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:38:22.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345976 -lt 85899345976 2024-06-15T10:38:22.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:22.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-06-15T10:38:22.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:22.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:38:22.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-06-15T10:38:22.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:22.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-06-15T10:38:22.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-06-15T10:38:22.784 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182448 2024-06-15T10:38:22.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:38:23.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-06-15T10:38:23.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:23.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:23.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018913 2024-06-15T10:38:23.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:38:23.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018913 2024-06-15T10:38:23.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:23.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018913 2024-06-15T10:38:23.079 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.5 seq 128849018913 2024-06-15T10:38:23.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018913' 2024-06-15T10:38:23.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:38:23.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018913 -lt 128849018913 2024-06-15T10:38:23.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:23.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-06-15T10:38:23.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:23.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:38:23.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-06-15T10:38:23.345 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:23.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-06-15T10:38:23.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-06-15T10:38:23.346 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.6 seq 150323855387 2024-06-15T10:38:23.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:38:23.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855387 -lt 150323855387 2024-06-15T10:38:23.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:23.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:23.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691860 2024-06-15T10:38:23.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:38:23.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691860 2024-06-15T10:38:23.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691860 2024-06-15T10:38:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2024-06-15T10:38:23.609 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.7 seq 171798691860 2024-06-15T10:38:23.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:38:23.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691860 -lt 171798691860 2024-06-15T10:38:23.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:23.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528337 2024-06-15T10:38:23.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:23.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:38:23.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528337 2024-06-15T10:38:23.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528337 2024-06-15T10:38:23.859 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.8 seq 193273528337 2024-06-15T10:38:23.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-06-15T10:38:23.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:38:24.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528337 -lt 193273528337 2024-06-15T10:38:24.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:24.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397509 2024-06-15T10:38:24.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:24.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:38:24.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397509 2024-06-15T10:38:24.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:24.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-06-15T10:38:24.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397509' 2024-06-15T10:38:24.110 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.9 seq 210453397509 2024-06-15T10:38:24.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:38:24.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397509 -lt 210453397509 2024-06-15T10:38:24.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:38:24.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:24.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:24.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:38:24.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:38:24.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:38:24.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:38:24.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:38:24.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:24.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:25.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:38:25.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-06-15T10:38:25.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:38:25.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:38:26.409 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:38:26.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:38:27.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-15T10:38:27.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-15T10:38:28.298 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 allow_ec_overwrites to true 2024-06-15T10:38:28.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:38:28.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:38:28.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:38:28.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:38:28.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:38:28.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:38:28.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:38:28.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:38:28.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:38:28.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:38:28.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:38:28.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:38:28.718 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:28.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:38:28.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-06-15T10:38:28.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-06-15T10:38:28.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-06-15T10:38:28.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:28.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:38:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673028 2024-06-15T10:38:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673028 2024-06-15T10:38:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028' 2024-06-15T10:38:29.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.015 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:38:29.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-06-15T10:38:29.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-06-15T10:38:29.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503' 2024-06-15T10:38:29.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:38:29.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345978 2024-06-15T10:38:29.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345978 2024-06-15T10:38:29.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978' 2024-06-15T10:38:29.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:38:29.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-06-15T10:38:29.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-06-15T10:38:29.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451' 2024-06-15T10:38:29.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.582 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:38:29.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018916 2024-06-15T10:38:29.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018916 2024-06-15T10:38:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451 5-128849018916' 2024-06-15T10:38:29.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:38:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-06-15T10:38:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-06-15T10:38:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451 5-128849018916 6-150323855390' 2024-06-15T10:38:29.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:29.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691863 2024-06-15T10:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691863 2024-06-15T10:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451 5-128849018916 6-150323855390 7-171798691863' 2024-06-15T10:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:30.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:38:30.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-06-15T10:38:30.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-06-15T10:38:30.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451 5-128849018916 6-150323855390 7-171798691863 8-193273528340' 2024-06-15T10:38:30.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:30.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:38:30.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397512 2024-06-15T10:38:30.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397512 2024-06-15T10:38:30.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673028 2-64424509503 3-85899345978 4-107374182451 5-128849018916 6-150323855390 7-171798691863 8-193273528340 9-210453397512' 2024-06-15T10:38:30.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:30.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-06-15T10:38:30.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:30.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:38:30.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-06-15T10:38:30.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:30.304 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836558 2024-06-15T10:38:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-06-15T10:38:30.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-06-15T10:38:30.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:38:30.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-06-15T10:38:30.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:30.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673028 2024-06-15T10:38:30.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:30.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:38:30.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673028 2024-06-15T10:38:30.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:30.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673028 2024-06-15T10:38:30.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-06-15T10:38:30.564 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949673028 2024-06-15T10:38:30.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:38:30.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673028 -lt 42949673028 2024-06-15T10:38:30.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:30.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-06-15T10:38:30.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:30.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:38:30.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:30.847 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-06-15T10:38:30.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-06-15T10:38:30.848 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509503 2024-06-15T10:38:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-06-15T10:38:30.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:38:31.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-06-15T10:38:31.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:31.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345978 2024-06-15T10:38:31.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:31.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:38:31.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345978 2024-06-15T10:38:31.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:31.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345978 2024-06-15T10:38:31.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-06-15T10:38:31.115 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.3 seq 85899345978 2024-06-15T10:38:31.115 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:38:31.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345978 -lt 85899345978 2024-06-15T10:38:31.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:31.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-06-15T10:38:31.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:31.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:38:31.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-06-15T10:38:31.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:31.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-06-15T10:38:31.380 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.4 seq 107374182451 2024-06-15T10:38:31.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-06-15T10:38:31.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:38:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182451 2024-06-15T10:38:31.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:31.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018916 2024-06-15T10:38:31.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:31.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:38:31.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018916 2024-06-15T10:38:31.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018916 2024-06-15T10:38:31.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-06-15T10:38:31.652 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.5 seq 128849018916 2024-06-15T10:38:31.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:38:31.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018916 2024-06-15T10:38:31.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:31.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:31.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-06-15T10:38:31.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:38:31.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-06-15T10:38:31.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:31.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-06-15T10:38:31.889 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.6 seq 150323855390 2024-06-15T10:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-06-15T10:38:31.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:38:32.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855390 -lt 150323855390 2024-06-15T10:38:32.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:32.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691863 2024-06-15T10:38:32.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:32.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:38:32.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691863 2024-06-15T10:38:32.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:32.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691863 2024-06-15T10:38:32.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691863' 2024-06-15T10:38:32.128 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.7 seq 171798691863 2024-06-15T10:38:32.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:38:32.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691863 -lt 171798691863 2024-06-15T10:38:32.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:32.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-06-15T10:38:32.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:32.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:38:32.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-06-15T10:38:32.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:32.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-06-15T10:38:32.394 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.8 seq 193273528340 2024-06-15T10:38:32.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-06-15T10:38:32.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:38:32.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-06-15T10:38:32.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:32.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397512 2024-06-15T10:38:32.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:32.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:38:32.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397512 2024-06-15T10:38:32.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:32.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397512 2024-06-15T10:38:32.638 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.9 seq 210453397512 2024-06-15T10:38:32.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397512' 2024-06-15T10:38:32.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:38:32.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397512 -lt 210453397512 2024-06-15T10:38:32.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:38:32.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:32.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:33.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:38:33.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:38:33.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:38:33.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:38:33.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:38:33.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:38:33.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:38:33.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:38:33.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:38:33.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:38:33.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:33.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T10:38:33.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:38:33.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:38:34.122 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:38:34.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:38:34.490 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:38:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:38:34.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:38:34.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:38:35.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-06-15T10:38:35.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-06-15T10:38:35.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:38:35.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:38:35.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:38:35.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-06-15T10:38:35.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-06-15T10:38:35.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-06-15T10:38:35.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T10:38:35.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-06-15T10:38:35.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:38:35.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:38:35.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:9 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr:7' 2024-06-15T10:38:35.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-06-15T10:38:35.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-06-15T10:38:35.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:38:35.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:38:35.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:38:35.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:38:35.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:38:35.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:38:35.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:38:35.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:38:35.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:38:35.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:38:35.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:38:35.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:38:35.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:38:38.357 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:38:38.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:38:38.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:38:38.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:38:38.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:38:38.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:38:38.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:38:38.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:38:38.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:38:38.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:38:38.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:38:38.901 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.3 2024-06-15T10:38:38.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:38:38.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:38:38.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:38:38.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:38:38.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:38:38.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:38:38.934 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:38.933+0000 7f0f574fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:38.937 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:38.937+0000 7f0f574fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:38.938 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:38.937+0000 7f0f574fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:39.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:38:39.152 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:38:39.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:39.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:38:39.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:39.489 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:39.489+0000 7f0f574fe7c0 -1 Falling back to public interface 2024-06-15T10:38:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:38:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:40.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:40.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:38:40.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:41.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:38:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:41.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:38:41.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:42.259 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:42.261+0000 7f0f574fe7c0 -1 osd.3 67 log_to_monitors true 2024-06-15T10:38:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:38:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:38:42.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:42.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:38:43.410 INFO:tasks.workunit.client.0.smithi116.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/3082961315,v1:127.0.0.1:6827/3082961315] [v2:127.0.0.1:6828/3082961315,v1:127.0.0.1:6829/3082961315] exists,up a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:38:43.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:38:43.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:38:43.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:38:43.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:38:43.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:43.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:38:44.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-06-15T10:38:44.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-06-15T10:38:44.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-06-15T10:38:44.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:38:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-06-15T10:38:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-06-15T10:38:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032' 2024-06-15T10:38:44.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:38:44.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-06-15T10:38:44.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-06-15T10:38:44.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507' 2024-06-15T10:38:44.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-06-15T10:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-06-15T10:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018' 2024-06-15T10:38:44.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:38:44.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182455 2024-06-15T10:38:44.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182455 2024-06-15T10:38:44.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455' 2024-06-15T10:38:44.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:38:44.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018920 2024-06-15T10:38:44.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018920 2024-06-15T10:38:44.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455 5-128849018920' 2024-06-15T10:38:44.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:44.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:38:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-06-15T10:38:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-06-15T10:38:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455 5-128849018920 6-150323855394' 2024-06-15T10:38:45.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:45.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:38:45.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-06-15T10:38:45.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-06-15T10:38:45.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455 5-128849018920 6-150323855394 7-171798691867' 2024-06-15T10:38:45.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:45.258 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:38:45.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-06-15T10:38:45.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-06-15T10:38:45.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455 5-128849018920 6-150323855394 7-171798691867 8-193273528344' 2024-06-15T10:38:45.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:38:45.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:38:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397516 2024-06-15T10:38:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397516 2024-06-15T10:38:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509507 3-304942678018 4-107374182455 5-128849018920 6-150323855394 7-171798691867 8-193273528344 9-210453397516' 2024-06-15T10:38:45.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:45.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-06-15T10:38:45.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:45.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:38:45.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-06-15T10:38:45.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-06-15T10:38:45.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-06-15T10:38:45.552 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836562 2024-06-15T10:38:45.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:38:45.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836562 2024-06-15T10:38:45.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:45.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-06-15T10:38:45.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:45.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:38:45.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-06-15T10:38:45.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:45.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-06-15T10:38:45.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-06-15T10:38:45.781 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673032 2024-06-15T10:38:45.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:38:46.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-06-15T10:38:46.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:46.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-06-15T10:38:46.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:46.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:38:46.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-06-15T10:38:46.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:46.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-06-15T10:38:46.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-06-15T10:38:46.056 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509507 2024-06-15T10:38:46.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:38:46.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509507 2024-06-15T10:38:46.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:46.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-06-15T10:38:46.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:46.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:38:46.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-06-15T10:38:46.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:46.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-06-15T10:38:46.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-06-15T10:38:46.346 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 304942678018 2024-06-15T10:38:46.346 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:38:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-06-15T10:38:46.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:46.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182455 2024-06-15T10:38:46.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:46.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:38:46.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:46.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182455 2024-06-15T10:38:46.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182455 2024-06-15T10:38:46.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-06-15T10:38:46.607 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182455 2024-06-15T10:38:46.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:38:46.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182455 -lt 107374182455 2024-06-15T10:38:46.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:46.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018920 2024-06-15T10:38:46.861 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:46.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:38:46.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018920 2024-06-15T10:38:46.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:46.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018920 2024-06-15T10:38:46.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-06-15T10:38:46.865 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 128849018920 2024-06-15T10:38:46.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:38:47.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018920 2024-06-15T10:38:47.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:47.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:47.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-06-15T10:38:47.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:38:47.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-06-15T10:38:47.121 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-06-15T10:38:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-06-15T10:38:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855394 2024-06-15T10:38:47.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:38:47.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855394 -lt 150323855394 2024-06-15T10:38:47.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:47.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691867 2024-06-15T10:38:47.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:47.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:38:47.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691867 2024-06-15T10:38:47.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:47.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-06-15T10:38:47.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-06-15T10:38:47.368 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691867 2024-06-15T10:38:47.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:38:47.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691867 2024-06-15T10:38:47.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:47.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-06-15T10:38:47.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:47.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:38:47.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-06-15T10:38:47.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:47.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-06-15T10:38:47.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-06-15T10:38:47.613 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528344 2024-06-15T10:38:47.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:38:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528344 -lt 193273528344 2024-06-15T10:38:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:38:47.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397516 2024-06-15T10:38:47.849 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:38:47.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:38:47.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397516 2024-06-15T10:38:47.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:38:47.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397516 2024-06-15T10:38:47.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397516' 2024-06-15T10:38:47.853 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 210453397516 2024-06-15T10:38:47.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:38:48.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397516 -lt 210453397516 2024-06-15T10:38:48.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:38:48.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:48.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:38:48.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:38:48.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:38:48.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:38:48.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:38:48.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:38:48.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:38:48.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:38:48.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:38:48.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:38:49.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:38:49.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:38:49.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:38:49.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:38:49.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:38:49.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:38:49.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:38:49.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:38:49.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:49.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:38:49.625 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:38:49.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:38:49.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:38:49.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:38:49.643 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:38:49.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:26.194115+0000 '>' 2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:49.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:38:50.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:38:50.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:38:50.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:38:50.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:38:50.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:38:50.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:38:50.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:38:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:26.194115+0000 '>' 2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:38:52.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:38:52.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:38:52.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:38:52.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:38:52.113 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:38:52.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:38:52.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:38:52.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:26.194115+0000 '>' 2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:52.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:38:53.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:38:53.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:38:53.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:38:53.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:38:53.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:38:53.361 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:38:53.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:50.283168+0000 '>' 2024-06-15T10:38:26.194115+0000 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:38:53.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:38:53.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:38:53.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:38:53.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:38:53.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:38:53.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:38:56.071 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:38:56.071 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:38:56.071 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:38:56.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:38:56.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:38:56.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:38:56.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:38:56.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:38:56.607 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:38:56.609 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:38:56.609 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:38:56.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:38:56.611 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.9 2024-06-15T10:38:56.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:38:56.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:38:56.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:38:56.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:38:56.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:38:56.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:38:56.643 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:56.641+0000 7f8772bf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:56.646 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:56.645+0000 7f8772bf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:56.647 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:56.649+0000 7f8772bf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:38:56.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:38:56.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:38:56.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:38:56.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:56.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:57.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:57.193 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:57.193+0000 7f8772bf57c0 -1 Falling back to public interface 2024-06-15T10:38:58.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:58.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:58.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:38:58.152 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:38:58.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:58.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:58.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:59.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:38:59.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:38:59.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:38:59.395 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:38:59.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:38:59.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:38:59.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:38:59.846 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:38:59.845+0000 7f8772bf57c0 -1 osd.9 72 log_to_monitors true 2024-06-15T10:39:00.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:00.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:00.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:39:00.645 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:00.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:00.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [49,72) [v2:127.0.0.1:6874/61348140,v1:127.0.0.1:6875/61348140] [v2:127.0.0.1:6876/61348140,v1:127.0.0.1:6877/61348140] exists,up 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:39:00.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:39:00.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:39:00.892 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:39:00.892 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:39:00.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:39:00.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:39:00.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:39:01.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:39:01.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:39:01.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:39:01.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:39:01.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:39:01.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:39:01.672 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:01.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:39:02.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-06-15T10:39:02.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-06-15T10:39:02.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-06-15T10:39:02.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:02.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:39:02.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-06-15T10:39:02.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-06-15T10:39:02.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036' 2024-06-15T10:39:02.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:02.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:39:02.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-06-15T10:39:02.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-06-15T10:39:02.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512' 2024-06-15T10:39:02.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:02.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:39:02.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678023 2024-06-15T10:39:02.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678023 2024-06-15T10:39:02.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023' 2024-06-15T10:39:02.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:02.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:39:02.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182459 2024-06-15T10:39:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182459 2024-06-15T10:39:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459' 2024-06-15T10:39:02.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:02.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:39:03.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-06-15T10:39:03.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-06-15T10:39:03.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459 5-128849018925' 2024-06-15T10:39:03.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:03.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:39:03.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-06-15T10:39:03.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-06-15T10:39:03.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459 5-128849018925 6-150323855399' 2024-06-15T10:39:03.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:03.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:39:03.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691871 2024-06-15T10:39:03.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691871 2024-06-15T10:39:03.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459 5-128849018925 6-150323855399 7-171798691871' 2024-06-15T10:39:03.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:03.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:39:03.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-06-15T10:39:03.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-06-15T10:39:03.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459 5-128849018925 6-150323855399 7-171798691871 8-193273528348' 2024-06-15T10:39:03.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:03.424 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:39:03.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-06-15T10:39:03.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-06-15T10:39:03.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673036 2-64424509512 3-304942678023 4-107374182459 5-128849018925 6-150323855399 7-171798691871 8-193273528348 9-330712481794' 2024-06-15T10:39:03.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:03.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-06-15T10:39:03.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:03.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:39:03.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-06-15T10:39:03.567 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:03.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-06-15T10:39:03.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-06-15T10:39:03.568 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836567 2024-06-15T10:39:03.568 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:39:03.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836567 2024-06-15T10:39:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:03.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-06-15T10:39:03.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:03.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:39:03.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-06-15T10:39:03.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:03.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-06-15T10:39:03.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-06-15T10:39:03.823 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673036 2024-06-15T10:39:03.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:39:04.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673037 -lt 42949673036 2024-06-15T10:39:04.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:04.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-06-15T10:39:04.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:04.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:39:04.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-06-15T10:39:04.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:04.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-06-15T10:39:04.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-06-15T10:39:04.047 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509512 2024-06-15T10:39:04.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:39:04.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-06-15T10:39:04.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:04.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678023 2024-06-15T10:39:04.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:04.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:04.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678023 2024-06-15T10:39:04.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678023 2024-06-15T10:39:04.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-06-15T10:39:04.287 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 304942678023 2024-06-15T10:39:04.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:39:04.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678023 2024-06-15T10:39:04.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:04.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182459 2024-06-15T10:39:04.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:04.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:39:04.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182459 2024-06-15T10:39:04.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:04.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182459 2024-06-15T10:39:04.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-06-15T10:39:04.540 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182459 2024-06-15T10:39:04.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:39:04.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182459 2024-06-15T10:39:04.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:04.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:04.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-06-15T10:39:04.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:39:04.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-06-15T10:39:04.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:04.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-06-15T10:39:04.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-06-15T10:39:04.820 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 128849018925 2024-06-15T10:39:04.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:39:05.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018925 2024-06-15T10:39:05.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:05.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-06-15T10:39:05.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:05.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:39:05.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-06-15T10:39:05.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:05.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-06-15T10:39:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-06-15T10:39:05.046 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855399 2024-06-15T10:39:05.046 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:39:05.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855399 2024-06-15T10:39:05.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:05.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691871 2024-06-15T10:39:05.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:05.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:39:05.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691871 2024-06-15T10:39:05.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:05.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691871 2024-06-15T10:39:05.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691871' 2024-06-15T10:39:05.286 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691871 2024-06-15T10:39:05.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:39:05.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691871 -lt 171798691871 2024-06-15T10:39:05.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:05.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-06-15T10:39:05.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:05.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:39:05.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-06-15T10:39:05.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:05.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-06-15T10:39:05.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-06-15T10:39:05.525 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528348 2024-06-15T10:39:05.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:39:05.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528348 2024-06-15T10:39:05.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:05.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-06-15T10:39:05.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:05.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:39:05.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-06-15T10:39:05.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:05.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-06-15T10:39:05.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-06-15T10:39:05.789 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 330712481794 2024-06-15T10:39:05.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:39:06.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-06-15T10:39:06.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:39:06.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:06.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:06.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:39:06.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:39:06.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:39:06.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:39:06.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:39:06.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:06.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:06.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:39:06.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:39:06.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:39:06.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:39:06.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:06.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:39:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:39:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:39:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:39:07.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:39:09.332 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:39:09.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:39:09.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:39:09.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.5 2024-06-15T10:39:09.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:39:09.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:39:09.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:39:09.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:39:09.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:39:09.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:39:09.923 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:09.920+0000 7fb24af877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:09.925 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:09.924+0000 7fb24af877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:09.927 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:09.924+0000 7fb24af877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:10.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:10.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:10.462 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:10.460+0000 7fb24af877c0 -1 Falling back to public interface 2024-06-15T10:39:11.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:11.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:11.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:39:11.389 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:39:11.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:11.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:11.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:12.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:12.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:12.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:39:12.641 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:39:12.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:12.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:12.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:13.248+0000 7fb24af877c0 -1 osd.5 79 log_to_monitors true 2024-06-15T10:39:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:39:13.876 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:osd.5 up in weight 1 up_from 83 up_thru 68 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/3892898621,v1:127.0.0.1:6843/3892898621] [v2:127.0.0.1:6844/3892898621,v1:127.0.0.1:6845/3892898621] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:39:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:39:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:39:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:39:14.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:39:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:39:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:39:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:39:14.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:39:14.220 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:39:14.221 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:39:14.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:39:14.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:39:14.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:39:14.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:39:14.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:39:14.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:14.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:39:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-06-15T10:39:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-06-15T10:39:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-06-15T10:39:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:14.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:39:15.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-06-15T10:39:15.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-06-15T10:39:15.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040' 2024-06-15T10:39:15.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:15.311 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:39:15.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-06-15T10:39:15.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-06-15T10:39:15.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515' 2024-06-15T10:39:15.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:15.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:39:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-06-15T10:39:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-06-15T10:39:15.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026' 2024-06-15T10:39:15.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:15.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:39:15.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-06-15T10:39:15.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-06-15T10:39:15.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463' 2024-06-15T10:39:15.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:15.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:39:16.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-06-15T10:39:16.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-06-15T10:39:16.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463 5-356482285570' 2024-06-15T10:39:16.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:16.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:39:16.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855402 2024-06-15T10:39:16.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855402 2024-06-15T10:39:16.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463 5-356482285570 6-150323855402' 2024-06-15T10:39:16.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:16.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:39:16.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691875 2024-06-15T10:39:16.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691875 2024-06-15T10:39:16.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463 5-356482285570 6-150323855402 7-171798691875' 2024-06-15T10:39:16.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:16.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:39:16.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528352 2024-06-15T10:39:16.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528352 2024-06-15T10:39:16.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463 5-356482285570 6-150323855402 7-171798691875 8-193273528352' 2024-06-15T10:39:16.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:16.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:39:16.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-06-15T10:39:16.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-06-15T10:39:16.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673040 2-64424509515 3-304942678026 4-107374182463 5-356482285570 6-150323855402 7-171798691875 8-193273528352 9-330712481798' 2024-06-15T10:39:16.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:16.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-06-15T10:39:16.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:16.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:39:16.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-06-15T10:39:16.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:16.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-06-15T10:39:16.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-06-15T10:39:16.719 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836571 2024-06-15T10:39:16.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:39:16.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-06-15T10:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-06-15T10:39:16.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:16.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:39:16.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-06-15T10:39:16.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:16.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-06-15T10:39:16.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-06-15T10:39:16.955 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673040 2024-06-15T10:39:16.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:39:17.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-06-15T10:39:17.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:17.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:17.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-06-15T10:39:17.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:39:17.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-06-15T10:39:17.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:17.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-06-15T10:39:17.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-06-15T10:39:17.176 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509515 2024-06-15T10:39:17.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:39:17.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509515 2024-06-15T10:39:17.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:17.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:17.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-06-15T10:39:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:39:17.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-06-15T10:39:17.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:17.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-06-15T10:39:17.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-06-15T10:39:17.457 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 304942678026 2024-06-15T10:39:17.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:39:17.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678027 -lt 304942678026 2024-06-15T10:39:17.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:17.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-06-15T10:39:17.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:17.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:39:17.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-06-15T10:39:17.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:17.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-06-15T10:39:17.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-06-15T10:39:17.796 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182463 2024-06-15T10:39:17.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:39:18.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182463 2024-06-15T10:39:18.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:18.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285570 2024-06-15T10:39:18.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:18.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:39:18.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285570 2024-06-15T10:39:18.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:18.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-06-15T10:39:18.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-06-15T10:39:18.051 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 356482285570 2024-06-15T10:39:18.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:39:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-06-15T10:39:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:18.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855402 2024-06-15T10:39:18.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:18.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:39:18.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855402 2024-06-15T10:39:18.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:18.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855402 2024-06-15T10:39:18.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-06-15T10:39:18.286 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855402 2024-06-15T10:39:18.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:39:18.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855402 2024-06-15T10:39:18.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:18.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:18.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691875 2024-06-15T10:39:18.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:39:18.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691875 2024-06-15T10:39:18.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:18.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691875 2024-06-15T10:39:18.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-06-15T10:39:18.553 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691875 2024-06-15T10:39:18.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:39:18.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691875 -lt 171798691875 2024-06-15T10:39:18.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:18.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:18.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528352 2024-06-15T10:39:18.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:39:18.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528352 2024-06-15T10:39:18.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528352 2024-06-15T10:39:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-06-15T10:39:18.842 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528352 2024-06-15T10:39:18.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:39:19.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528352 2024-06-15T10:39:19.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:19.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-06-15T10:39:19.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:19.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:39:19.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-06-15T10:39:19.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:19.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-06-15T10:39:19.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-06-15T10:39:19.089 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 330712481798 2024-06-15T10:39:19.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:39:19.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-06-15T10:39:19.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:39:19.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:19.371 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:19.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:39:19.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:39:19.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:39:19.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:39:19.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:39:19.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:39:19.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:39:19.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:39:19.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:39:19.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:39:19.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:19.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:20.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:39:20.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:39:20.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:39:20.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-06-15T10:39:20.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:39:20.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:39:20.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:39:20.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:39:20.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-06-15T10:39:20.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-06-15T10:39:20.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:39:20.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:39:20.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:20.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:20.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:20.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:20.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:20.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:39:20.878 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:39:20.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:20.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:20.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:20.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:20.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:20.897 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:21.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:50.283168+0000 '>' 2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:21.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:39:22.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:39:22.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:22.123 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:22.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:22.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:22.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:22.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:22.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:38:50.283168+0000 '>' 2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:22.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:39:23.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:39:23.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:23.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:23.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:23.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:23.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:23.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:23.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:21.205205+0000 '>' 2024-06-15T10:38:50.283168+0000 2024-06-15T10:39:23.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:39:23.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:39:23.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:39:23.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:39:23.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:39:23.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:39:26.052 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:39:26.052 INFO:tasks.workunit.client.0.smithi116.stdout:hinfo_key 2024-06-15T10:39:26.052 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:39:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:39:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:39:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:39:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:39:26.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:39:26.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:39:26.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:39:26.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:39:26.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:39:26.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:39:26.594 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.9 2024-06-15T10:39:26.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:39:26.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:39:26.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:39:26.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:39:26.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:39:26.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:39:26.627 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:26.624+0000 7f382f4207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:26.629 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:26.628+0000 7f382f4207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:26.631 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:26.628+0000 7f382f4207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:26.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:27.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:27.174 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:27.172+0000 7f382f4207c0 -1 Falling back to public interface 2024-06-15T10:39:28.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:28.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:28.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:39:28.116 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:39:28.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:28.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:28.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:29.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:39:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:39:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:29.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:29.473 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:39:29.472+0000 7f382f4207c0 -1 osd.9 85 log_to_monitors true 2024-06-15T10:39:29.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:39:30.694 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:30.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:osd.9 up in weight 1 up_from 89 up_thru 0 down_at 86 last_clean_interval [77,85) [v2:127.0.0.1:6874/14303820,v1:127.0.0.1:6875/14303820] [v2:127.0.0.1:6876/14303820,v1:127.0.0.1:6877/14303820] exists,up 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:39:30.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:39:31.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:39:31.000 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:39:31.000 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:39:31.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:39:31.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:39:31.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:39:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:39:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:39:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:39:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:39:31.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:39:31.114 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:4 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:5 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:6 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:7 2024-06-15T10:39:31.732 INFO:tasks.workunit.client.0.smithi116.stderr:8 2024-06-15T10:39:31.733 INFO:tasks.workunit.client.0.smithi116.stderr:9' 2024-06-15T10:39:31.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:39:31.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:31.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:39:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-06-15T10:39:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-06-15T10:39:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-06-15T10:39:31.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:31.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:39:32.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-06-15T10:39:32.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-06-15T10:39:32.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044' 2024-06-15T10:39:32.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:32.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:39:32.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-06-15T10:39:32.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-06-15T10:39:32.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520' 2024-06-15T10:39:32.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:32.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:39:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678031 2024-06-15T10:39:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678031 2024-06-15T10:39:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031' 2024-06-15T10:39:32.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:32.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:39:33.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-06-15T10:39:33.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-06-15T10:39:33.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467' 2024-06-15T10:39:33.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:33.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:39:33.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285574 2024-06-15T10:39:33.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285574 2024-06-15T10:39:33.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467 5-356482285574' 2024-06-15T10:39:33.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:33.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:39:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855407 2024-06-15T10:39:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855407 2024-06-15T10:39:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467 5-356482285574 6-150323855407' 2024-06-15T10:39:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:33.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:39:33.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-06-15T10:39:33.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-06-15T10:39:33.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467 5-356482285574 6-150323855407 7-171798691879' 2024-06-15T10:39:33.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:33.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:39:33.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-06-15T10:39:33.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-06-15T10:39:33.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467 5-356482285574 6-150323855407 7-171798691879 8-193273528356' 2024-06-15T10:39:33.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:33.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:39:33.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-06-15T10:39:33.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-06-15T10:39:33.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673044 2-64424509520 3-304942678031 4-107374182467 5-356482285574 6-150323855407 7-171798691879 8-193273528356 9-382252089346' 2024-06-15T10:39:33.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:33.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-06-15T10:39:33.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:33.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:39:33.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-06-15T10:39:33.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:33.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-06-15T10:39:33.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-06-15T10:39:33.771 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836575 2024-06-15T10:39:33.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:39:34.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836575 2024-06-15T10:39:34.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:34.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-06-15T10:39:34.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:34.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:39:34.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-06-15T10:39:34.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:34.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-06-15T10:39:34.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-06-15T10:39:34.008 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949673044 2024-06-15T10:39:34.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:39:34.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673044 2024-06-15T10:39:34.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:34.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-06-15T10:39:34.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:34.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:39:34.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-06-15T10:39:34.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:34.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-06-15T10:39:34.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-06-15T10:39:34.236 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509520 2024-06-15T10:39:34.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:39:34.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-06-15T10:39:34.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:34.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678031 2024-06-15T10:39:34.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:34.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:39:34.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678031 2024-06-15T10:39:34.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:34.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678031 2024-06-15T10:39:34.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678031' 2024-06-15T10:39:34.499 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.3 seq 304942678031 2024-06-15T10:39:34.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:39:34.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678031 -lt 304942678031 2024-06-15T10:39:34.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:34.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-06-15T10:39:34.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:34.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:39:34.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-06-15T10:39:34.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:34.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-06-15T10:39:34.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-06-15T10:39:34.754 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.4 seq 107374182467 2024-06-15T10:39:34.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:39:34.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182467 2024-06-15T10:39:34.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:34.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285574 2024-06-15T10:39:34.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:34.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:39:34.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285574 2024-06-15T10:39:34.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:34.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285574 2024-06-15T10:39:34.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285574' 2024-06-15T10:39:34.983 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.5 seq 356482285574 2024-06-15T10:39:34.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:39:35.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285574 -lt 356482285574 2024-06-15T10:39:35.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:35.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855407 2024-06-15T10:39:35.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:35.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:39:35.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855407 2024-06-15T10:39:35.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:35.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855407 2024-06-15T10:39:35.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-06-15T10:39:35.223 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.6 seq 150323855407 2024-06-15T10:39:35.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:39:35.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855407 -lt 150323855407 2024-06-15T10:39:35.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:35.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-06-15T10:39:35.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:35.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:39:35.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-06-15T10:39:35.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-06-15T10:39:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-06-15T10:39:35.468 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.7 seq 171798691879 2024-06-15T10:39:35.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:39:35.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-06-15T10:39:35.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:35.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-06-15T10:39:35.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:35.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:39:35.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:35.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-06-15T10:39:35.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-06-15T10:39:35.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-06-15T10:39:35.709 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.8 seq 193273528356 2024-06-15T10:39:35.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:39:35.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528357 -lt 193273528356 2024-06-15T10:39:35.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:35.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-382252089346 2024-06-15T10:39:35.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:35.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:39:35.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-382252089346 2024-06-15T10:39:35.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-06-15T10:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 382252089346' 2024-06-15T10:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.9 seq 382252089346 2024-06-15T10:39:35.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:39:36.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089346 2024-06-15T10:39:36.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:39:36.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:36.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:39:36.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:39:36.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:39:36.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:39:36.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:39:36.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:36.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:36.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:39:36.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:39:36.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:39:36.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:39:37.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:39:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 270569"' 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 270569' 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:39:37.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 270570"' 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 270570' 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 270569 270570' 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 270569 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/270572: /' 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:39:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/270574: /' 2024-06-15T10:39:40.430 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:39:40.431 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:39:40.432 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:39:40.433 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:39:40.436 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:39:40.436 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:39:40.436 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:39:40.436 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:39:40.437 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: start osd.5 2024-06-15T10:39:40.438 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:39:40.539 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-06-15T10:39:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:270574: remove 2#2:eb822e21:::SOMETHING:head# 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:39:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:39:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:39:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: start osd.9 2024-06-15T10:39:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:39:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:39:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:39:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:39:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:39:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2024-06-15T10:39:40.468+0000 7f9d760627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2024-06-15T10:39:40.472+0000 7f9d760627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2024-06-15T10:39:40.472+0000 7f9d760627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 0 2024-06-15T10:39:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2024-06-15T10:39:41.060+0000 7f9d760627c0 -1 Falling back to public interface 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 1 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2024-06-15T10:39:44.040+0000 7f9d760627c0 -1 osd.5 91 log_to_monitors true 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:39:44.753 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 3 2024-06-15T10:39:44.754 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:44.754 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270572: osd.5 up in weight 1 up_from 95 up_thru 83 down_at 92 last_270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2024-06-15T10:39:40.584+0000 7f8a48b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2024-06-15T10:39:40.588+0000 7f8a48b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2024-06-15T10:39:40.588+0000 7f8a48b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 0 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2024-06-15T10:39:41.148+0000 7f8a48b1d7c0 -1 Falling back to public interface 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 1 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:45.178 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2024-06-15T10:39:44.312+0000 7f8a48b1d7c0 -1 osd.9 91 log_to_monitors true 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 3 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:45.179 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:46.328 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [83,91) [v2:127.0.0.1:6842/913795924,v1:127.0.0.1:6843/913795924] [v2:127.0.0.1:6844/913795924,v1:127.0.0.1:6845/913795924] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:39:46.329 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 1 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 2 2024-06-15T10:39:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 3 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 4 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 5 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 6 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 7 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 8 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: 9' 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:39:46.331 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048' 2024-06-15T10:39:46.332 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/sts.sh:985: wait_for_osd: sleep 1 2024-06-15T10:39:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:39:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:39:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:39:47.309 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 4 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6874/486487693,v1:127.0.0.1:6875/486487693] [v2:127.0.0.1:6876/486487693,v1:127.0.0.1:6877/486487693] exists,up 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:39:47.310 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:39:47.311 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 1 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 2 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 3 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 4 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 5 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 6 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 7 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 8 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: 9' 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.312 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:39:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-06-15T10:39:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-06-15T10:39:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-06-15T10:39:47.313 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.951 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //homandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:39:47.951 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524' 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678034 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678034 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034' 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471' 2024-06-15T10:39:47.952 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471 5-408021893122' 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471 5-408021893122 6-150323855411' 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-06-15T10:39:47.953 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-06-15T10:39:47.954 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471 5-408021893122 6-150323855411 7-171798691883' 2024-06-15T10:39:47.954 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:47.954 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.cle/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049' 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.517 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525' 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678036 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678036 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036' 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-06-15T10:39:48.518 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472' 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472 5-408021893123' 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472 5-408021893123 6-150323855412' 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.519 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_sient.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528360 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528360 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471 5-408021893122 6-150323855411 7-171798691883 8-193273528360' 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673048 2-64424509524 3-304942678034 4-107374182471 5-408021893122 6-150323855411 7-171798691883 8-193273528360 9-412316860418' 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-06-15T10:39:48.544 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.0 seq 21474836579 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836579 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:48.545 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.1 seq 42949673048 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:48.546 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.clietats: seq=171798691884 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472 5-408021893123 6-150323855412 7-171798691884' 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472 5-408021893123 6-150323855412 7-171798691884 8-193273528361' 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:39:49.052 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673049 2-64424509525 3-304942678036 4-107374182472 5-408021893123 6-150323855412 7-171798691884 8-193273528361 9-412316860419' 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.0 seq 21474836580 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:39:49.053 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836580 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-06-15T10:39:49.054 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.1 seq 42949673049 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_statsnt.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.2 seq 64424509524 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509524 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678034 2024-06-15T10:39:49.314 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678034 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678034 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678034' 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.3 seq 304942678034 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678035 -lt 304942678034 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:39:49.315 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.4 seq 107374182471 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182471 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893122 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893122 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:49.316 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-06-15T10:39:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2024-06-15T10:39:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.5 seq 408021893122 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.6 seq 150323855411 2024-06-15T10:39:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855412 -lt 150323855411 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.7 seq 171798691883 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-06-15T10:39:50.269 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528360 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528360 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528360 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.8 seq 193273528360 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:39:50.270 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528361 -lt 193273528360 2024-06-15T10:39:51.110 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_st: ceph osd last-stat-seq 1 2024-06-15T10:39:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673049 2024-06-15T10:39:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:39:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:39:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:39:51.111 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673049 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-06-15T10:39:51.112 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.2 seq 64424509525 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-06-15T10:39:51.113 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678036 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678036 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:51.114 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678036 2024-06-15T10:39:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678036' 2024-06-15T10:39:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.3 seq 304942678036 2024-06-15T10:39:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:39:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678036 -lt 304942678036 2024-06-15T10:39:51.115 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-06-15T10:39:51.116 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-06-15T10:39:51.117 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.4 seq 107374182472 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.cliats: for s in $seqs 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-06-15T10:39:51.354 INFO:tasks.workunit.client.0.smithi116.stderr:270572: waiting osd.9 seq 412316860418 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:39:51.355 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:270572: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:39:51.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 270570 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:ent.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182472 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893123 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893123 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2024-06-15T10:39:52.168 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.5 seq 408021893123 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.6 seq 150323855412 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:39:52.169 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855412 -lt 150323855412 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.7 seq 171798691884 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691885 -lt 171798691884 2024-06-15T10:39:52.170 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:53.224 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.8 seq 193273528361 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528361 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:39:53.225 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860419 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860419 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: waiting osd.9 seq 412316860419 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:53.226 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:39:53.227 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:39:53.228 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:39:53.228 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:39:53.228 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:39:53.484 INFO:tasks.workunit.client.0.smithi116.stderr:270574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:270574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:39:53.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:39:53.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:39:53.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:39:53.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:39:53.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:39:53.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:39:53.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:39:53.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:39:53.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:39:53.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:39:53.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:53.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:53.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:53.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:53.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:53.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:39:54.209 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:39:54.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:54.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:39:54.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:54.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:39:54.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:39:54.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:54.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:54.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:54.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:54.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:54.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:54.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:21.205205+0000 '>' 2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:54.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:39:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:39:55.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:55.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:21.205205+0000 '>' 2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:55.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:39:56.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:39:56.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:56.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:56.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:56.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:56.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:56.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:56.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:21.205205+0000 '>' 2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:56.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:39:57.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:39:57.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:39:57.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:39:57.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:39:57.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:39:57.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:39:58.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:54.977694+0000 '>' 2024-06-15T10:39:21.205205+0000 2024-06-15T10:39:58.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 273997"' 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 273997' 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 273999"' 2024-06-15T10:39:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/274000: /' 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 273999' 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 273997 273999' 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 273997 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:39:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/274003: /' 2024-06-15T10:40:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: _ 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: hinfo_key 2024-06-15T10:40:01.779 INFO:tasks.workunit.client.0.smithi116.stderr:274000: snapset 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:01.780 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:01.782 INFO:tasks.workunit.client.0.smithi116.stderr:274000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:01.783 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:01.784 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:40:01.784 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:40:01.784 INFO:tasks.workunit.client.0.smithi116.stderr:274000: start osd.5 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-06-15T10:40:02.328 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:02.329 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: _ 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: hinfo_key 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: snapset 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:02.330 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:02.331 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:02.332 INFO:tasks.workunit.client.0.smithi116.stderr:274003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-06-15T10:40:02.333 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-06-15T10:40:02.334 INFO:tasks.workunit.client.0.smithi116.stderr:274003: start osd.9 2024-06-15T10:40:06.179 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:40:06.179 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:40:06.179 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2024-06-15T10:40:01.815+0000 7fe8d43787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2024-06-15T10:40:01.815+0000 7fe8d43787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2024-06-15T10:40:01.819+0000 7fe8d43787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 0 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2024-06-15T10:40:02.391+0000 7fe8d43787c0 -1 Falling back to public interface 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 1 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2024-06-15T10:40:05.327+0000 7fe8d43787c0 -1 osd.5 98 log_to_monitors true 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 3 2024-06-15T10:40:06.182 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.183 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:06.614 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/covery_ops 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2024-06-15T10:40:02.359+0000 7f4463d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2024-06-15T10:40:02.359+0000 7f4463d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2024-06-15T10:40:02.359+0000 7f4463d8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:06.615 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 0 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2024-06-15T10:40:02.903+0000 7f4463d8f7c0 -1 Falling back to public interface 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.616 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 1 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2024-06-15T10:40:05.999+0000 7f4463d8f7c0 -1 osd.9 98 log_to_monitors true 2024-06-15T10:40:06.617 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:06.618 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:06.618 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:06.618 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 3 2024-06-15T10:40:06.618 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:06.618 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 4 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-06-15T10:40:08.644 INFO:tasks.workunit.client.0.smithi116.stderr:274003: osd.9 up in weight 1 up_from 106 up_thru 0 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6874/3564499840,v1:127.0.0.1:6875/3564499840] [v2:127.0.0.1:6876/3564499840,v1:127.0.0.1:6877/3564499840] exists,up 6a7107f6-73ef-4a7a-b9fd-ea42fb5206f3 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:08.645 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 1 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 2 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 3 2024-06-15T10:40:08.646 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 4 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 5 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 6 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 7 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 8 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: 9' 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ideph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:08.647 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 4 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: osd.5 up in weight 1 up_from 103 up_thru 103 down_at 99 last_clean_interval [95,98) [v2:127.0.0.1:6842/1150762600,v1:127.0.0.1:6843/1150762600] [v2:127.0.0.1:6844/1150762600,v1:127.0.0.1:6845/1150762600] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:08.648 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:08.649 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 1 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 2 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 3 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 4 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 5 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 6 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 7 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 8 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: 9' 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:08.650 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:08.651 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-06-15T10:40:08.651 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-06-15T10:40:08.651 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-06-15T10:40:10.265 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056' 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530' 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.266 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678041 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678041 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041' 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478' 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-06-15T10:40:10.267 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478 5-442381631491' 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478 5-442381631491 6-150323855417' 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shs 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055' 2024-06-15T10:40:10.268 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531' 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042' 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-06-15T10:40:10.269 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479' 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479 5-442381631490' 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479 5-442381631490 6-150323855418' 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.270 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:10.806 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2:2202: flush_pg_stats: seq=171798691890 2024-06-15T10:40:10.806 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-06-15T10:40:10.806 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478 5-442381631491 6-150323855417 7-171798691890' 2024-06-15T10:40:10.806 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478 5-442381631491 6-150323855417 7-171798691890 8-193273528367' 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509530 3-304942678041 4-107374182478 5-442381631491 6-150323855417 7-171798691890 8-193273528367 9-455266533378' 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:10.807 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.0 seq 21474836586 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:10.808 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.1 seq 42949673056 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221202: flush_pg_stats: seq=171798691891 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691891 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479 5-442381631490 6-150323855418 7-171798691891' 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.809 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479 5-442381631490 6-150323855418 7-171798691891 8-193273528368' 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533379 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533379 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673055 2-64424509531 3-304942678042 4-107374182479 5-442381631490 6-150323855418 7-171798691891 8-193273528368 9-455266533379' 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:10.810 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.0 seq 21474836585 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836585 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-06-15T10:40:10.811 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:10.812 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-06-15T10:40:10.812 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-06-15T10:40:10.812 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.1 seq 42949673055 2024-06-15T10:40:12.773 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214:4: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:12.773 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673056 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.2 seq 64424509530 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509530 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.774 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678041 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678041 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678041 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678041' 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.3 seq 304942678041 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678040 -lt 304942678041 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:12.775 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678042 -lt 304942678041 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.4 seq 107374182478 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673055 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.776 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.2 seq 64424509531 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:12.777 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.3 seq 304942678042 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678040 -lt 304942678042 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678042 -lt 304942678042 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-06-15T10:40:12.778 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-06-15T10:40:12.779 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.4 seq 107374182479 2024-06-15T10:40:13.829 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182479 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631490 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631490 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 442381631490' 2024-06-15T10:40:13.830 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.5 seq 442381631490 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.6 seq 150323855418 2024-06-15T10:40:13.831 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855418 -lt 150323855418 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691891 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691891 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691891 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.7 seq 171798691891 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691891 -lt 171798691891 2024-06-15T10:40:13.832 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:13.835 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182478 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631491 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631491 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 442381631491' 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.5 seq 442381631491 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.836 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.6 seq 150323855417 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855418 -lt 150323855417 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:13.837 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.7 seq 171798691890 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691891 -lt 171798691890 2024-06-15T10:40:13.838 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:14.867 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-06-15T10:40:14.867 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:14.867 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:14.867 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-06-15T10:40:14.867 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.8 seq 193273528367 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528367 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533378 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533378 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-06-15T10:40:14.868 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533378' 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: waiting osd.9 seq 455266533378 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533378 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:14.869 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: chelpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:14.870 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.8 seq 193273528368 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528368 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533379 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533379 2024-06-15T10:40:14.871 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533379 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533379' 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: waiting osd.9 seq 455266533379 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533379 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:14.872 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:40:14.873 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:15.939 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:15.940 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:40:15.941 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:40:15.942 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:40:15.942 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:40:15.942 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:40:15.942 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:40:15.942 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:40:15.976 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_ceph --format json status 2024-06-15T10:40:15.976 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:15.976 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:15.977 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:40:15.978 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:40:15.979 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:40:15.979 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:40:16.619 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_forlean: (( 1 >= 13 )) 2024-06-15T10:40:16.619 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:40:16.619 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:16.620 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:16.621 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:16.621 INFO:tasks.workunit.client.0.smithi116.stderr:274003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:16.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:16.663 INFO:tasks.workunit.client.0.smithi116.stderr:_clean: (( 1 >= 13 )) 2024-06-15T10:40:16.663 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:40:16.663 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:40:16.663 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:40:16.663 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:16.664 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:274000: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 273999 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:40:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:40:16.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:40:16.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 277568"' 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 277568' 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 277569"' 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 277569' 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 277568 277569' 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:40:16.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:40:16.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 277568 2024-06-15T10:40:16.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/277571: /' 2024-06-15T10:40:16.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:40:16.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/277573: /' 2024-06-15T10:40:20.706 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:40:20.707 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:20.708 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: remove 1#2:eb822e21:::SOMETHING:head# 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:20.709 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:40:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: start osd.5 2024-06-15T10:40:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:40:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:40:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:20.843 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:20.844 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-06-15T10:40:20.845 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.846 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:20.846 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:40:20.846 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:20.846 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:40:20.846 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: remove 0#2:eb822e21:::SOMETHING:head# 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:40:20.847 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:40:20.848 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:20.849 INFO:tasks.workunit.client.0.smithi116.stderr:277571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:20.850 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:20.851 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:40:20.851 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:40:20.851 INFO:tasks.workunit.client.0.smithi116.stderr:277571: start osd.3 2024-06-15T10:40:20.851 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:40:25.172 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:40:25.172 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2024-06-15T10:40:20.743+0000 7f46a29db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2024-06-15T10:40:20.747+0000 7f46a29db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2024-06-15T10:40:20.747+0000 7f46a29db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:25.173 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 0 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2024-06-15T10:40:21.311+0000 7f46a29db7c0 -1 Falling back to public interface 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 1 2024-06-15T10:40:25.174 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2024-06-15T10:40:24.551+0000 7f46a29db7c0 -1 osd.5 108 log_to_monitors true 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.175 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 3 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2024-06-15T10:40:20.871+0000 7f825d3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2024-06-15T10:40:20.875+0000 7f825d3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.176 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2024-06-15T10:40:20.875+0000 7f825d3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 0 2024-06-15T10:40:25.177 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2024-06-15T10:40:21.411+0000 7f825d3ec7c0 -1 Falling back to public interface 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 1 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.178 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2024-06-15T10:40:24.827+0000 7f825d3ec7c0 -1 osd.3 108 log_to_monitors true 2024-06-15T10:40:25.179 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 3 2024-06-15T10:40:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 4 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: osd.5 up in weight 1 up_from 114 up_thru 114 down_at 109 last_clean_interval [103,108) [v2:127.0.0.1:6826/540462654,v1:127.0.0.1:6827/540462654] [v2:127.0.0.1:6828/540462654,v1:127.0.0.1:6829/540462654] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:40:27.386 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:27.387 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 1 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 2 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 3 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 4 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 5 2024-06-15T10:40:27.388 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 6 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 7 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 8 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: 9' 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-06-15T10:40:27.389 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277573rs.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 4 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 109 last_clean_interval [71,108) [v2:127.0.0.1:6842/795666443,v1:127.0.0.1:6843/795666443] [v2:127.0.0.1:6844/795666443,v1:127.0.0.1:6845/795666443] exists,up a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:27.390 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:27.391 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 1 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 2 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 3 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 4 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 5 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 6 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 7 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 8 2024-06-15T10:40:27.392 INFO:tasks.workunit.client.0.smithi116.stderr:277571: 9' 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592' 2024-06-15T10:40:27.393 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:277571:: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-06-15T10:40:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-06-15T10:40:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061' 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537' 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747' 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182485 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182485 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485' 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485 5-489626271747' 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-06-15T10:40:28.797 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485 5-489626271747 6-150323855423' 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flu //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060' 2024-06-15T10:40:28.798 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536' 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746' 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:28.799 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182484 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182484 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484' 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484 5-489626271746' 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484 5-489626271746 6-150323855424' 2024-06-15T10:40:28.800 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:28.801 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691896 2024-06-15T10:40:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-06-15T10:40:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484 5-489626271746 6-150323855424 7-171798691896' 2024-06-15T10:40:29.287 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528373 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528373 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484 5-489626271746 6-150323855424 7-171798691896 8-193273528373' 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533384 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533384 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509536 3-489626271746 4-107374182484 5-489626271746 6-150323855424 7-171798691896 8-193273528373 9-455266533384' 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:29.288 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.0 seq 21474836592 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836592 -lt 21474836592 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-06-15T10:40:29.289 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:29.290 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-06-15T10:40:29.290 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-06-15T10:40:29.290 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.1 seq 42949673060 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pgsh_pg_stats: seq=171798691895 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485 5-489626271747 6-150323855423 7-171798691895' 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485 5-489626271747 6-150323855423 7-171798691895 8-193273528372' 2024-06-15T10:40:29.296 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673061 2-64424509537 3-489626271747 4-107374182485 5-489626271747 6-150323855423 7-171798691895 8-193273528372 9-455266533385' 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-06-15T10:40:29.297 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.0 seq 21474836591 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836592 -lt 21474836591 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-06-15T10:40:29.298 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.1 seq 42949673061 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673061 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:30.392 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.2 seq 64424509537 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271747 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271747 2024-06-15T10:40:30.393 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.3 seq 489626271747 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271747 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182485 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182485 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182485 2024-06-15T10:40:30.394 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.4 seq 107374182485 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182485 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-489626271747 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673060 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.2 seq 64424509536 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271746 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271746 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.3 seq 489626271746 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271746 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182484 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182484 2024-06-15T10:40:30.397 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182484 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.4 seq 107374182484 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182484 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:30.398 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-489626271746 2024-06-15T10:40:31.132 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu//cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-489626271747 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 489626271747' 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.5 seq 489626271747 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271747 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.133 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.6 seq 150323855423 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855423 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-06-15T10:40:31.134 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.7 seq 171798691895 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691895 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-06-15T10:40:31.135 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:31.136 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats:cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-489626271746 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 489626271746' 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.5 seq 489626271746 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:31.208 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271746 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.6 seq 150323855424 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855424 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.209 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.7 seq 171798691896 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691896 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528373 2024-06-15T10:40:31.210 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:31.211 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:31.211 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528373 2024-06-15T10:40:32.519 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:32.519 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-06-15T10:40:32.519 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-06-15T10:40:32.519 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.8 seq 193273528372 2024-06-15T10:40:32.519 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528372 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533385 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533385 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533385' 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: waiting osd.9 seq 455266533385 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:32.520 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:32.521 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:277573: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:32.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:32.545 INFO:tasks.workunit.client.0.smithi116.stderr: cut -d - -f 2 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528373 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.8 seq 193273528373 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528373 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533384 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:32.546 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533384 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533384 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533384' 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: waiting osd.9 seq 455266533384 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533384 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:32.547 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:32.548 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:277571: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 277569 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:40:32.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-06-15T10:40:32.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:40:32.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:40:32.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-06-15T10:40:32.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:32.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:32.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:32.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:33.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:33.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T10:40:33.322 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-06-15T10:40:33.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T10:40:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:40:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:40:33.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:40:33.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:40:33.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:33.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:33.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:33.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:33.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:54.977694+0000 '>' 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:33.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:40:34.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:40:34.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:40:34.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:40:34.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:34.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:34.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:34.592 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:34.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:54.977694+0000 '>' 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:34.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:40:35.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:40:35.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:40:35.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:40:35.839 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:35.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:35.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:35.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:36.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:54.977694+0000 '>' 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:36.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:40:37.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:40:37.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:40:37.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:40:37.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:37.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:37.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:37.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:39:54.977694+0000 '>' 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:40:38.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:40:38.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:40:38.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T10:40:38.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T10:40:38.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:40:38.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:40:38.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:40:33.831290+0000 '>' 2024-06-15T10:39:54.977694+0000 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:40:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 281024"' 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 281024' 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 281026"' 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 281026' 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 281024 281026' 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-06-15T10:40:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:40:38.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 281024 2024-06-15T10:40:38.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/281027: /' 2024-06-15T10:40:38.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:38.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/281029: /' 2024-06-15T10:40:42.634 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:42.635 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:42.635 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-06-15T10:40:42.635 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:40:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-06-15T10:40:42.636 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:42.637 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-06-15T10:40:42.637 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:42.637 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:42.637 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:42.638 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:42.639 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: _ 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: hinfo_key 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: snapset 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-06-15T10:40:42.640 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:42.641 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:42.642 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:42.643 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:42.643 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:42.643 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-06-15T10:40:42.643 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-06-15T10:40:42.643 INFO:tasks.workunit.client.0.smithi116.stderr:281027: start osd.3 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:40:43.417 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:40:43.418 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: _ 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: hinfo_key 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: snapset 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:43.419 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:43.421 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:43.422 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-06-15T10:40:43.423 INFO:tasks.workunit.client.0.smithi116.stderr:281029: start osd.5 2024-06-15T10:40:46.712 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:40:46.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-06-15T10:40:46.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-06-15T10:40:46.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2024-06-15T10:40:42.675+0000 7fc2803577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2024-06-15T10:40:42.731+0000 7fc2803577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2024-06-15T10:40:42.735+0000 7fc2803577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-06-15T10:40:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 0 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2024-06-15T10:40:43.267+0000 7fc2803577c0 -1 Falling back to public interface 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 1 2024-06-15T10:40:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2024-06-15T10:40:46.167+0000 7fc2803577c0 -1 osd.3 115 log_to_monitors true 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:46.716 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 3 2024-06-15T10:40:46.716 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:40:48.013 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2024-06-15T10:40:43.455+0000 7f2edfd087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2024-06-15T10:40:43.459+0000 7f2edfd087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2024-06-15T10:40:43.463+0000 7f2edfd087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:40:48.014 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 0 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2024-06-15T10:40:44.011+0000 7f2edfd087c0 -1 Falling back to public interface 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 1 2024-06-15T10:40:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2024-06-15T10:40:47.447+0000 7f2edfd087c0 -1 osd.5 115 log_to_monitors true 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 3 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:48.354 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-06-15T10:40:48.354 INFO:tasks.workunit.client.0.smithi116.stderr:281027: osd.3 up in weight 1 up_from 120 up_thru 114 down_at 116 last_clean_interval [114,115) [v2:127.0.0.1:6826/4117555597,v1:127.0.0.1:6827/4117555597] [v2:127.0.0.1:6828/4117555597,v1:127.0.0.1:6829/4117555597] exists,up a51f6c65-8c89-467b-b441-fcc49e4149e0 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:48.355 INFO:tasks.workunit.client.0.smithi116.stderr:281027: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 1 2024-06-15T10:40:48.356 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 2 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 3 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 4 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 5 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 6 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 7 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 8 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: 9' 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:48.357 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836597 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836597 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597' 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673067 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673067 2024-06-15T10:40:48.358 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067' 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 4 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: osd.5 up in weight 1 up_from 123 up_thru 123 down_at 116 last_clean_interval [114,115) [v2:127.0.0.1:6842/4054987928,v1:127.0.0.1:6843/4054987928] [v2:127.0.0.1:6844/4054987928,v1:127.0.0.1:6845/4054987928] exists,up 907a0d98-b3d5-4486-b9bb-78712b04c347 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:40:50.568 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:40:50.569 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 1 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 2 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 3 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 4 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 5 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 6 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 7 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 8 2024-06-15T10:40:50.570 INFO:tasks.workunit.client.0.smithi116.stderr:281029: 9' 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836599 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836599 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599' 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd /qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:50.571 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509542 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509542 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542' 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522' 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:50.572 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182490 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182490 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490' 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490 5-528280977410' 2024-06-15T10:40:50.573 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855429 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855429 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490 5-528280977410 6-150323855429' 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691902 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691902 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490 5-528280977410 6-150323855429 7-171798691902' 2024-06-15T10:40:50.574 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528379 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528379 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490 5-528280977410 6-150323855429 7-171798691902 8-193273528379' 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533390 2024-06-15T10:40:51.965 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533390 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673067 2-64424509542 3-515396075522 4-107374182490 5-528280977410 6-150323855429 7-171798691902 8-193273528379 9-455266533390' 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836597 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836597 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836597 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.0 seq 21474836597 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:51.966 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836597 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673067 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673067 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673067 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.1 seq 42949673067 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673067 -lt 42949673067 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:51.967 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509542 2024-06-15T10:40:51.968 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.118 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cepin $ids 2024-06-15T10:40:52.118 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:40:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673068 2024-06-15T10:40:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673068 2024-06-15T10:40:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068' 2024-06-15T10:40:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509544 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509544 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544' 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-06-15T10:40:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523' 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182491 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182491 2024-06-15T10:40:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491' 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977411 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977411 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491 5-528280977411' 2024-06-15T10:40:52.122 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491 5-528280977411 6-150323855430' 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.123 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-06-15T10:40:52.728 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509542 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509542 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.2 seq 64424509542 2024-06-15T10:40:52.729 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509544 -lt 64424509542 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075522 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075522 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075522 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075522' 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.3 seq 515396075522 2024-06-15T10:40:52.730 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182490 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182490 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182490 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.4 seq 107374182490 2024-06-15T10:40:52.731 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182490 -lt 107374182490 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977410 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977410 2024-06-15T10:40:52.732 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.757 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5282s.sh:2202: flush_pg_stats: seq=171798691903 2024-06-15T10:40:52.757 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491 5-528280977411 6-150323855430 7-171798691903' 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528380 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528380 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491 5-528280977411 6-150323855430 7-171798691903 8-193273528380' 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533392 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533392 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836599 1-42949673068 2-64424509544 3-515396075523 4-107374182491 5-528280977411 6-150323855430 7-171798691903 8-193273528380 9-455266533392' 2024-06-15T10:40:52.758 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836599 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836599 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836599 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.0 seq 21474836599 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836599 -lt 21474836599 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673068 2024-06-15T10:40:52.759 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673068 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673068 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-06-15T10:40:52.760 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.1 seq 42949673068 2024-06-15T10:40:53.709 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh80977410 2024-06-15T10:40:53.709 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977410' 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.5 seq 528280977410 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855429 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855429 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855429 2024-06-15T10:40:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.6 seq 150323855429 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855429 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691902 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691902 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691902 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.7 seq 171798691902 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691902 -lt 171798691902 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528379 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528379 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528379 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-06-15T10:40:53.712 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.8 seq 193273528379 2024-06-15T10:40:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673068 -lt 42949673068 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509544 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:40:53.855 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509544 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509544 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.2 seq 64424509544 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509544 -lt 64424509544 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-515396075523 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-515396075523 2024-06-15T10:40:53.856 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 515396075523' 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.3 seq 515396075523 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182491 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182491 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182491 2024-06-15T10:40:53.857 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-06-15T10:40:53.858 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.4 seq 107374182491 2024-06-15T10:40:53.858 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-06-15T10:40:53.858 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182492 -lt 107374182491 2024-06-15T10:40:53.858 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:53.858 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977411 2024-06-15T10:40:54.667 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977411 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977411 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977411' 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.5 seq 528280977411 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977411 -lt 528280977411 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:54.668 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.6 seq 150323855430 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855431 -lt 150323855430 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-06-15T10:40:54.669 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.7 seq 171798691903 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691903 -lt 171798691903 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528380 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-06-15T10:40:54.670 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528380 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: f93273528379 -lt 193273528379 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533390 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:54.688 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533390 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533390 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533390' 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: waiting osd.9 seq 455266533390 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533392 -lt 455266533390 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:54.689 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:281027: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-06-15T10:40:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 281026 2024-06-15T10:40:55.886 INFO:tasks.workunit.client.0.smithi116.stderr:lush_pg_stats: cut -d - -f 2 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528380 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528380' 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.8 seq 193273528380 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528380 -lt 193273528380 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-455266533392 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-455266533392 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533392 2024-06-15T10:40:55.887 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 455266533392' 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: waiting osd.9 seq 455266533392 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533392 -lt 455266533392 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:40:55.888 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:281029: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-06-15T10:40:55.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-06-15T10:40:55.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-06-15T10:40:55.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-06-15T10:40:55.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:40:55.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:40:55.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:40:55.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:40:55.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:40:55.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:40:55.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:55.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:55.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:55.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:55.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:56.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:56.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:40:56.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:40:56.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:40:56.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:40:56.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:40:56.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:40:56.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:40:56.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:40:56.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:40:56.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:40:56.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:40:56.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:40:56.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:40:56.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:40:56.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:56.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:56.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:40:56.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:40:56.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:40:56.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:40:56.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:40:56.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:40:56.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:40:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:40:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:40:56.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:40:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:40:56.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:40:56.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:40:56.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:40:56.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:40:56.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:40:56.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:40:56.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:40:56.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:56.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:56.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:40:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:40:56.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:40:56.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:40:56.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:40:56.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:56.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:56.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:40:56.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:40:56.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:40:56.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:40:56.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:40:56.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:40:56.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:40:56.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:40:56.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:40:56.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-15T10:40:56.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:40:56.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:56.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:56.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:56.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:56.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:56.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:56.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-15T10:40:56.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:40:56.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:40:56.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:40:56.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:40:56.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:40:56.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:40:56.990 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:40:56.990 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:40:56.990 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:40:56.991 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:40:56.991 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:56.992 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:40:56.992 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:40:56.992 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:40:57.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:40:57.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:40:57.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:40:57.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:40:57.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:40:57.106 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:40:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:40:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:40:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:40:57.107 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:40:57.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:40:57.108 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:40:57.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:40:58.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:40:58.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:58.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:58.098 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:58.099 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:58.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:58.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:40:58.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:40:58.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:40:58.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:40:58.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:40:58.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:40:58.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:40:58.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 2024-06-15T10:40:58.139 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 2024-06-15T10:40:58.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87' 2024-06-15T10:40:58.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:40:58.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6b21mydd5CRAAcs5XcsNuQsmJZC5UrT9BwQ== 2024-06-15T10:40:58.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6b21mydd5CRAAcs5XcsNuQsmJZC5UrT9BwQ=="}' 2024-06-15T10:40:58.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:40:58.409 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:40:58.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:40:58.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6b21mydd5CRAAcs5XcsNuQsmJZC5UrT9BwQ== --osd-uuid 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 2024-06-15T10:40:58.465 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.466+0000 7f0ec59377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:58.467 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.466+0000 7f0ec59377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:58.468 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.470+0000 7f0ec59377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:40:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.470+0000 7f0ec59377c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:40:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.470+0000 7f0ec59377c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:40:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.470+0000 7f0ec59377c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:40:58.551 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:40:58.554+0000 7f0ec59377c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:41:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:41:04.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:41:04.463 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:41:04.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:41:04.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:41:04.747 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:41:04.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:41:04.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:41:04.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:41:04.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:41:04.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:41:04.773 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:04.774+0000 7f3b8c75e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:04.775 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:04.778+0000 7f3b8c75e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:04.777 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:04.778+0000 7f3b8c75e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:05.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:41:05.029 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:41:05.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:05.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:05.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:05.326 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:05.326+0000 7f3b8c75e7c0 -1 Falling back to public interface 2024-06-15T10:41:06.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:06.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:06.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:41:06.289 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:41:06.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:06.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:06.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:07.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:07.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:07.532 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:41:07.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:41:07.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:07.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:07.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:07.834+0000 7f3b8c75e7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:41:08.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:08.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:08.778 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:41:08.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:41:08.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:08.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:09.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:10.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:10.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:10.019 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:41:10.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:41:10.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:10.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:10.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:11.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:11.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:11.250 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:41:11.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:41:11.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:11.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:11.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:12.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:12.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:12.618 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:41:12.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:41:12.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:12.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:12.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:13.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:13.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:13.860 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:41:13.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:41:13.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:13.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:14.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:15.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:15.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:15.103 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:41:15.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:41:15.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:15.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:15.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:16.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:16.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:16.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:41:16.347 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:41:16.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:16.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:16.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:17.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:17.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:17.698 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:41:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:41:17.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:17.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:17.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:18.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:18.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:18.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:41:18.943 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:41:18.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:18.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:19.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:20.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:20.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:20.186 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:41:20.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:41:20.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:20.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:20.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:21.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:21.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:21.896 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:41:21.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:41:21.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:21.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:22.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:23.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:23.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:23.136 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:41:23.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:41:23.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:23.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:23.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:24.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:24.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:24.692 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:41:24.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:41:24.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:24.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:25.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:26.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:26.620 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:41:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:41:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:26.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:26.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:27.890 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:41:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:41:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:28.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:29.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:29.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:29.431 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:41:29.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:41:29.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:29.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:41:29.670 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/818896281,v1:127.0.0.1:6803/818896281] [v2:127.0.0.1:6804/818896281,v1:127.0.0.1:6805/818896281] exists,up 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:41:29.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:41:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:41:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:41:29.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:41:29.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:41:29.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 193981c3-685b-4102-89d3-4e77a796585e' 2024-06-15T10:41:29.679 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:41:29.679 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:41:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZb21mZTDOKRAAB0sEZkFGDrUg/clE1OscJw== 2024-06-15T10:41:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZb21mZTDOKRAAB0sEZkFGDrUg/clE1OscJw=="}' 2024-06-15T10:41:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 193981c3-685b-4102-89d3-4e77a796585e -i td/osd-scrub-repair/1/new.json 2024-06-15T10:41:30.038 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:41:30.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:41:30.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZb21mZTDOKRAAB0sEZkFGDrUg/clE1OscJw== --osd-uuid 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:41:30.094 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.094+0000 7f9a76e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:30.113 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.114+0000 7f9a76e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:30.115 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.114+0000 7f9a76e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:30.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.118+0000 7f9a76e8f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:41:30.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.118+0000 7f9a76e8f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:41:30.116 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.118+0000 7f9a76e8f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:41:30.189 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:30.190+0000 7f9a76e8f7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:41:35.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:41:35.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:41:35.998 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:41:35.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:41:35.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:41:36.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:41:36.268 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:41:36.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:41:36.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:41:36.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:41:36.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:41:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:36.302+0000 7f73eba257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:36.302 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:36.302+0000 7f73eba257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:36.303 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:36.306+0000 7f73eba257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:41:36.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:41:36.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:41:36.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:36.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:36.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:36.886 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:36.886+0000 7f73eba257c0 -1 Falling back to public interface 2024-06-15T10:41:37.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:37.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:37.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:41:37.768 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:41:37.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:37.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:38.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:39.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:39.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:39.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:41:39.008 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:41:39.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:39.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:39.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:41:39.154+0000 7f73eba257c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:41:39.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:40.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:40.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:40.255 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:41:40.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:41:40.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:40.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:40.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:41.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:41.497 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:41:41.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:41:41.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:41.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:41.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:42.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:42.737 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:41:42.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:41:42.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:42.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:43.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:44.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:44.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:44.055 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:41:44.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:41:44.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:44.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:44.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:45.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:45.303 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:41:45.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:41:45.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:45.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:45.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:41:46.551 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:41:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:46.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:46.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:47.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:47.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:47.830 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:41:47.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:41:47.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:47.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:48.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:49.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:49.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:49.211 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:41:49.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:41:49.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:49.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:49.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:50.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:50.595 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:41:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:41:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:50.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:50.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:51.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:51.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:51.841 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:41:51.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:41:51.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:51.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:52.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:53.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:53.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:53.211 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:41:53.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:41:53.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:53.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:53.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:54.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:54.690 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:41:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:41:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:54.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:55.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:56.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:56.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:56.102 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:41:56.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:41:56.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:56.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:56.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:57.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:57.864 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:41:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:41:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:41:58.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:41:59.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:41:59.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:41:59.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:41:59.608 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:41:59.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:41:59.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:00.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:01.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:01.129 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:42:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:42:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:01.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:02.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:02.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:02.641 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:42:02.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:42:02.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:02.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:02.881 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4112063444,v1:127.0.0.1:6811/4112063444] [v2:127.0.0.1:6812/4112063444,v1:127.0.0.1:6813/4112063444] exists,up 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:42:02.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:42:02.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:42:02.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:42:02.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-06-15T10:42:02.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:42:03.110 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:42:03.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:42:03.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:42:03.474 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:42:03.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:42:04.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:42:06.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-06-15T10:42:06.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:42:06.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:42:06.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:42:06.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:42:06.336 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:42:06.336 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:42:06.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:42:06.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:42:06.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:42:06.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:42:06.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:42:06.704 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:42:06.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:42:06.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:06.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:42:06.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-06-15T10:42:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-06-15T10:42:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-06-15T10:42:06.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:06.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:42:07.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705671 2024-06-15T10:42:07.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705671 2024-06-15T10:42:07.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-38654705671' 2024-06-15T10:42:07.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:07.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-06-15T10:42:07.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:07.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:42:07.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-06-15T10:42:07.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:07.008 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836494 2024-06-15T10:42:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-06-15T10:42:07.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-15T10:42:07.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-06-15T10:42:07.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:42:08.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:42:08.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:08.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-15T10:42:08.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:08.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705671 2024-06-15T10:42:08.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:08.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:42:08.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:08.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705671 2024-06-15T10:42:08.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705671 2024-06-15T10:42:08.480 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 38654705671 2024-06-15T10:42:08.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2024-06-15T10:42:08.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:42:08.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705672 -lt 38654705671 2024-06-15T10:42:08.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:42:08.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:08.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:08.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:42:08.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:42:08.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:42:09.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:42:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:42:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:09.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:42:09.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-06-15T10:42:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-06-15T10:42:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:42:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:42:09.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:42:09.807 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:42:09.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:42:10.201 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:42:10.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:42:10.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:42:10.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-06-15T10:42:10.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-06-15T10:42:10.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-06-15T10:42:10.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-06-15T10:42:10.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-06-15T10:42:10.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-15T10:42:10.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:42:10.451 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-15T10:42:10.451 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:42:10.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T10:42:10.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-06-15T10:42:10.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T10:42:10.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:42:10.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:42:10.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:42:11.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-06-15T10:42:12.738 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:42:13.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:42:13.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:42:13.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:42:13.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:42:13.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:42:13.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:42:13.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:42:13.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:42:13.325 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:13.325+0000 7fdfbef957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:13.327 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:13.329+0000 7fdfbef957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:13.329 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:13.329+0000 7fdfbef957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:13.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:42:13.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:13.862 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:13.861+0000 7fdfbef957c0 -1 Falling back to public interface 2024-06-15T10:42:14.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:14.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:14.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:42:14.779 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:42:14.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:14.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:42:15.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:42:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:42:16.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:16.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:42:16.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:16.524 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:16.513+0000 7fdfbef957c0 -1 osd.0 20 log_to_monitors true 2024-06-15T10:42:17.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:42:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:42:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:17.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:42:17.504 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/582670989,v1:127.0.0.1:6803/582670989] [v2:127.0.0.1:6804/582670989,v1:127.0.0.1:6805/582670989] exists,up 0db6dacd-a5c1-4c6d-9cde-59a2b5a3cd87 2024-06-15T10:42:17.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:42:17.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:42:17.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:42:17.506 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:42:17.506 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:42:17.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:42:17.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:42:17.506 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:42:17.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:42:17.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:42:17.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:42:17.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:42:17.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:42:17.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:42:17.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:42:17.849 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:42:17.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:42:17.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:17.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:42:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-06-15T10:42:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-06-15T10:42:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-06-15T10:42:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:18.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:42:18.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705675 2024-06-15T10:42:18.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705675 2024-06-15T10:42:18.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-38654705675' 2024-06-15T10:42:18.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:18.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-06-15T10:42:18.167 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:18.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:42:18.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-06-15T10:42:18.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:18.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-06-15T10:42:18.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-06-15T10:42:18.170 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182402 2024-06-15T10:42:18.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 107374182402 2024-06-15T10:42:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:42:19.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:42:19.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:19.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-06-15T10:42:19.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:19.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705675 2024-06-15T10:42:19.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:19.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:42:19.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705675 2024-06-15T10:42:19.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705675 2024-06-15T10:42:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2024-06-15T10:42:19.651 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 38654705675 2024-06-15T10:42:19.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:42:19.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705675 -lt 38654705675 2024-06-15T10:42:19.902 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:42:19.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:19.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:20.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:42:20.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:42:20.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:42:20.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:42:20.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:42:20.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:20.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:20.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:42:20.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:42:20.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:42:20.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-15T10:42:20.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-15T10:42:20.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:42:20.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:42:20.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-15T10:42:20.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-15T10:42:20.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-15T10:42:20.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-15T10:42:20.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-15T10:42:20.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:20.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:20.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:20.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:21.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:21.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-15T10:42:21.401 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-15T10:42:21.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:21.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-15T10:42:21.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:21.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:42:21.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:42:21.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:21.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:21.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:21.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:21.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:21.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:21.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:03.426220+0000 '>' 2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:21.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:22.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:22.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:22.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:22.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:22.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:22.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:22.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:22.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:03.426220+0000 '>' 2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:22.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:23.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:23.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:23.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:23.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:23.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:23.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:23.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:24.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:03.426220+0000 '>' 2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:24.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:25.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:25.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:25.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:25.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:25.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:25.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:25.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:25.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:22.204152+0000 '>' 2024-06-15T10:42:03.426220+0000 2024-06-15T10:42:25.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:42:25.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:25.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:42:25.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:42:25.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:42:25.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:42:26.842 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:42:26.843 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:42:26.843 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:42:27.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:42:27.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:42:27.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:42:27.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:42:27.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:42:27.151 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:42:27.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:42:27.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:42:27.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:42:27.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:42:27.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:42:27.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:42:27.184 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:27.185+0000 7f989a0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:27.187 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:27.185+0000 7f989a0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:27.189 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:27.189+0000 7f989a0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:42:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:42:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:42:27.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:27.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:27.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:27.749 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:27.749+0000 7f989a0c37c0 -1 Falling back to public interface 2024-06-15T10:42:28.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:28.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:28.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:42:28.681 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:42:28.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:28.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:28.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:29.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:29.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:29.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:42:29.926 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:42:29.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:29.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:30.087 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:30.089+0000 7f989a0c37c0 -1 osd.1 26 log_to_monitors true 2024-06-15T10:42:30.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:31.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:31.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:31.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:42:31.206 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:42:31.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:31.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [9,26) [v2:127.0.0.1:6810/783359430,v1:127.0.0.1:6811/783359430] [v2:127.0.0.1:6812/783359430,v1:127.0.0.1:6813/783359430] exists,up 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:42:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:42:31.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:42:31.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:42:31.486 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:42:31.486 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:42:31.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:42:31.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:42:31.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:42:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:42:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:42:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:42:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:42:31.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:42:31.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:42:31.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:42:31.836 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:42:31.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:42:31.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:31.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:42:32.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-06-15T10:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-06-15T10:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-06-15T10:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:32.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:42:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T10:42:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T10:42:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-128849018882' 2024-06-15T10:42:32.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:32.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-06-15T10:42:32.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:32.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:42:32.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-06-15T10:42:32.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:32.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-06-15T10:42:32.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-06-15T10:42:32.156 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182406 2024-06-15T10:42:32.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:32.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182406 2024-06-15T10:42:32.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:42:33.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:42:33.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:33.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-06-15T10:42:33.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:33.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-06-15T10:42:33.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:33.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:42:33.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-06-15T10:42:33.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T10:42:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-06-15T10:42:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 128849018882 2024-06-15T10:42:33.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:42:33.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T10:42:33.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:42:33.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:33.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:34.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:42:34.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:42:34.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:42:34.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:42:34.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:42:34.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:42:34.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:42:34.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:42:34.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:42:34.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:42:34.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:34.416 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:34.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:42:34.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:42:34.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:42:34.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:42:34.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:42:34.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-06-15T10:42:34.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-06-15T10:42:34.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-06-15T10:42:34.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-06-15T10:42:34.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:42:34.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:42:34.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:42:35.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:42:35.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-06-15T10:42:35.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:42:35.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:42:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:42:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:42:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-06-15T10:42:36.503 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-06-15T10:42:37.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:42:37.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:42:37.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:42:37.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:42:37.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:42:37.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:42:37.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:42:37.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:42:37.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:42:37.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:42:37.042 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:42:37.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:42:37.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:42:37.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:42:37.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:42:37.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:42:37.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:42:37.074 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:37.073+0000 7f6bef36a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:37.077 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:37.077+0000 7f6bef36a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:37.078 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:37.077+0000 7f6bef36a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:42:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:42:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:42:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:42:37.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:42:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:42:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:42:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:37.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:37.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:37.641 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:37.641+0000 7f6bef36a7c0 -1 Falling back to public interface 2024-06-15T10:42:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:42:38.594 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:42:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:38.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:39.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:39.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:39.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:42:39.837 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:42:39.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:39.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:40.060 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:40.061+0000 7f6bef36a7c0 -1 osd.1 31 log_to_monitors true 2024-06-15T10:42:40.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:41.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:42:41.080 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:42:41.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:41.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/1987953620,v1:127.0.0.1:6811/1987953620] [v2:127.0.0.1:6812/1987953620,v1:127.0.0.1:6813/1987953620] exists,up 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:42:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:42:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:42:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:42:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:42:41.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:42:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:42:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:42:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:42:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:42:41.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:42:41.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:42:41.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:42:41.667 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:42:41.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:42:41.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:41.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:42:41.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-06-15T10:42:41.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-06-15T10:42:41.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182409' 2024-06-15T10:42:41.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:41.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:42:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-06-15T10:42:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-06-15T10:42:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182409 1-150323855362' 2024-06-15T10:42:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:41.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182409 2024-06-15T10:42:41.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:41.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:42:41.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182409 2024-06-15T10:42:41.986 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:41.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-06-15T10:42:41.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182409' 2024-06-15T10:42:41.987 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182409 2024-06-15T10:42:41.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:42.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182409 2024-06-15T10:42:42.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:42:43.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:42:43.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:43.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-06-15T10:42:43.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:43.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:43.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-06-15T10:42:43.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:42:43.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-06-15T10:42:43.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-06-15T10:42:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-06-15T10:42:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 150323855362 2024-06-15T10:42:43.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:42:43.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-06-15T10:42:43.699 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:42:43.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:43.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:43.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:42:43.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:42:43.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:42:44.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:42:44.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:42:44.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:44.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:44.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:42:44.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:42:44.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:42:44.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-06-15T10:42:44.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-06-15T10:42:44.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-06-15T10:42:44.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-06-15T10:42:44.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:42:44.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-06-15T10:42:44.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-06-15T10:42:44.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-06-15T10:42:44.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-06-15T10:42:44.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:44.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:44.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:44.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:45.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:45.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-06-15T10:42:45.239 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.3 on osd.1 to repair 2024-06-15T10:42:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-06-15T10:42:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:42:45.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:42:45.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:45.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:45.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:45.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:45.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:45.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:45.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:22.204152+0000 '>' 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:45.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:46.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:46.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:46.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:22.204152+0000 '>' 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:46.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:47.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:47.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:47.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:47.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:47.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:47.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:47.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:47.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:22.204152+0000 '>' 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:47.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:48.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:48.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:48.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:48.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:48.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:48.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:49.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:22.204152+0000 '>' 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:49.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:42:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:42:50.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:42:50.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-06-15T10:42:50.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-06-15T10:42:50.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:42:50.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:42:50.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:42:46.105084+0000 '>' 2024-06-15T10:42:22.204152+0000 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:42:50.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:42:50.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:42:50.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:42:50.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:50.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-06-15T10:42:52.005 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-06-15T10:42:52.006 INFO:tasks.workunit.client.0.smithi116.stdout:_ 2024-06-15T10:42:52.006 INFO:tasks.workunit.client.0.smithi116.stdout:snapset 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:42:52.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:42:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:42:52.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:42:52.543 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:42:52.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:42:52.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:42:52.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:42:52.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:42:52.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:42:52.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:42:52.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:42:52.580 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:52.580+0000 7f53d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:52.599 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:52.596+0000 7f53d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:52.600 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:52.600+0000 7f53d0a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:52.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:53.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:53.142 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:53.140+0000 7f53d0a2b7c0 -1 Falling back to public interface 2024-06-15T10:42:54.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:54.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:54.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:42:54.051 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:42:54.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:54.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:54.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:55.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:55.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:55.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:42:55.289 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:42:55.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:55.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:55.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:42:55.696 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:42:55.696+0000 7f53d0a2b7c0 -1 osd.1 36 log_to_monitors true 2024-06-15T10:42:56.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:42:56.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:42:56.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:42:56.536 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:42:56.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:42:56.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:42:56.801 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [35,36) [v2:127.0.0.1:6810/1053623992,v1:127.0.0.1:6811/1053623992] [v2:127.0.0.1:6812/1053623992,v1:127.0.0.1:6813/1053623992] exists,up 193981c3-685b-4102-89d3-4e77a796585e 2024-06-15T10:42:56.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:42:56.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:42:56.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:42:56.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:42:56.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:42:56.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:42:56.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:42:56.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:42:56.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:42:57.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:42:57.142 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:42:57.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:42:57.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:57.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:42:57.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182413 2024-06-15T10:42:57.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182413 2024-06-15T10:42:57.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182413' 2024-06-15T10:42:57.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:42:57.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:42:57.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-06-15T10:42:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-06-15T10:42:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182413 1-171798691842' 2024-06-15T10:42:57.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:57.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182413 2024-06-15T10:42:57.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:57.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:42:57.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182413 2024-06-15T10:42:57.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:57.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182413 2024-06-15T10:42:57.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182413' 2024-06-15T10:42:57.511 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 107374182413 2024-06-15T10:42:57.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:57.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182411 -lt 107374182413 2024-06-15T10:42:57.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:42:58.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:42:58.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:42:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182413 -lt 107374182413 2024-06-15T10:42:58.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:42:58.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-06-15T10:42:58.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:42:58.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:42:58.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-06-15T10:42:58.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:42:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-06-15T10:42:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-06-15T10:42:58.984 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 171798691842 2024-06-15T10:42:58.984 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:42:59.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-06-15T10:42:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:42:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:42:59.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:59.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:42:59.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:42:59.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:42:59.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:42:59.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:42:59.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:42:59.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:42:59.508 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:42:59.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:42:59.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:42:59.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:42:59.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:43:00.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:43:00.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:43:00.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:43:00.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-06-15T10:43:00.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:43:00.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:43:00.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:43:00.168 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:43:00.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:43:00.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:43:00.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:43:00.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:43:00.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:43:00.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:43:00.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:43:00.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:43:00.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:43:00.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:43:00.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:43:00.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:43:00.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:43:00.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:43:00.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:43:00.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:43:00.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:43:00.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:43:00.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:43:00.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:43:00.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:43:00.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:43:00.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:43:00.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:43:00.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:43:00.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:43:00.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:43:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:43:00.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:43:00.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:43:00.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:43:00.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:43:00.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:43:00.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:43:00.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:43:00.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:43:00.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:43:00.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:43:00.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:43:00.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:43:00.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:43:00.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:43:00.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:43:00.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:43:00.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:43:00.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:43:00.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:43:00.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:43:00.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:43:00.526 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:43:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.527 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:00.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:43:00.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:43:00.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:43:00.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:43:00.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:43:00.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:43:00.753 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:43:00.753 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:43:00.753 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:43:00.753 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:43:00.754 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:43:00.754 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.754 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.754 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:43:00.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:43:00.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:43:00.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:43:00.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:43:00.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:43:00.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:43:00.873 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:43:00.874 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:43:00.874 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:00.874 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:00.875 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:43:00.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:43:00.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:43:00.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:43:00.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:01.868 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:01.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:01.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:43:01.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:43:01.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-15T10:43:01.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:43:01.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:01.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:43:01.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:43:01.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:43:01.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:43:01.915 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:43:01.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bcbf8693-030a-4230-97ca-85019753e5d6' 2024-06-15T10:43:01.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:43:01.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1cG1mp3mKNxAA91pr0ZegPhO0O89mqolqLg== 2024-06-15T10:43:01.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1cG1mp3mKNxAA91pr0ZegPhO0O89mqolqLg=="}' 2024-06-15T10:43:01.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcbf8693-030a-4230-97ca-85019753e5d6 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:43:02.178 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:43:02.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:43:02.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1cG1mp3mKNxAA91pr0ZegPhO0O89mqolqLg== --osd-uuid bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:43:02.232 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.232+0000 7f3a7e1a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:02.234 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.232+0000 7f3a7e1a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:02.235 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.236+0000 7f3a7e1a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:02.236 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.236+0000 7f3a7e1a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:43:02.236 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.236+0000 7f3a7e1a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:43:02.236 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.236+0000 7f3a7e1a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:43:02.313 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:02.312+0000 7f3a7e1a07c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:43:07.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:43:07.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:43:07.973 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:43:07.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:43:07.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:43:08.246 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:43:08.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:43:08.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:43:08.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:43:08.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:43:08.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:43:08.279 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:08.276+0000 7f26638457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:08.281 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:08.280+0000 7f26638457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:08.282 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:08.280+0000 7f26638457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:08.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:43:08.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:43:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:43:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:43:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:43:08.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:08.518 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:43:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:43:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:08.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:08.853 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:08.852+0000 7f26638457c0 -1 Falling back to public interface 2024-06-15T10:43:09.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:09.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:09.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:43:09.777 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:43:09.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:09.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:10.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:11.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:11.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:11.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:43:11.012 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:43:11.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:11.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:11.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:11.412 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:11.412+0000 7f26638457c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:43:12.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:12.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:12.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:43:12.244 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:43:12.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:12.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:12.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:13.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:13.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:13.487 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:43:13.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:43:13.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:13.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:14.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:14.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:14.729 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:43:14.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:43:14.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:14.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:16.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:16.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:16.057 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:43:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:43:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:16.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:17.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:17.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:17.365 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:43:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:43:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:17.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:17.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:18.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:18.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:18.611 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:43:18.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:43:18.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:18.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:18.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:19.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:19.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:19.850 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:43:19.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:43:19.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:19.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:20.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:21.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:21.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:21.226 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:43:21.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:43:21.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:21.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:21.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:22.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:22.470 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:43:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:43:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:22.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:22.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:23.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:23.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:23.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:43:23.715 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:43:23.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:23.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:24.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:25.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:25.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:25.393 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:43:25.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:43:25.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:25.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:25.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:26.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:26.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:26.719 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:43:26.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:43:26.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:26.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:27.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:28.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:28.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:28.478 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:43:28.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:43:28.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:28.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:28.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:29.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:29.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:29.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:43:29.992 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:43:29.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:29.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:30.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:31.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:31.609 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:43:31.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:43:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:31.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:32.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:33.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:33.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:33.381 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:43:33.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:43:33.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:33.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:43:33.630 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1167580621,v1:127.0.0.1:6803/1167580621] [v2:127.0.0.1:6804/1167580621,v1:127.0.0.1:6805/1167580621] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:43:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:43:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:43:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:43:33.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:43:33.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:43:33.634 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:43:33.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:43:33.637 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:43:33.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b645dff1-d4a4-407f-a604-f4118837f98f' 2024-06-15T10:43:33.638 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:43:33.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVcG1mjgp8JxAAoWnhXxxJ5DoLpUrq9wp6jA== 2024-06-15T10:43:33.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVcG1mjgp8JxAAoWnhXxxJ5DoLpUrq9wp6jA=="}' 2024-06-15T10:43:33.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b645dff1-d4a4-407f-a604-f4118837f98f -i td/osd-scrub-repair/1/new.json 2024-06-15T10:43:33.944 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:43:33.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:43:33.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVcG1mjgp8JxAAoWnhXxxJ5DoLpUrq9wp6jA== --osd-uuid b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:43:33.997 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:33.996+0000 7f68590417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:34.011 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.012+0000 7f68590417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:34.013 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.012+0000 7f68590417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:34.013 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.012+0000 7f68590417c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:43:34.013 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.012+0000 7f68590417c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:43:34.013 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.012+0000 7f68590417c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:43:34.080 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:34.080+0000 7f68590417c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:43:40.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:43:40.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:43:40.209 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:43:40.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:43:40.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:43:40.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:43:40.484 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:43:40.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:43:40.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:43:40.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:43:40.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:43:40.517 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:40.516+0000 7f2214d937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:40.558 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:40.556+0000 7f2214d937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:40.560 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:40.560+0000 7f2214d937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:43:40.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:43:40.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:43:40.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:43:40.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:43:40.750 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:43:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:43:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:43:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:40.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:40.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:41.106 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:41.104+0000 7f2214d937c0 -1 Falling back to public interface 2024-06-15T10:43:41.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:41.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:41.991 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:43:41.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:43:41.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:41.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:42.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:43.234 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:43:43.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:43:43.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:43.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:43.452 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:43:43.452+0000 7f2214d937c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:43:43.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:44.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:44.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:44.483 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:43:44.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:43:44.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:44.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:44.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:45.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:45.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:45.725 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:43:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:43:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:45.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:45.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:46.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:46.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:46.968 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:43:46.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:43:46.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:46.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:47.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:48.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:48.321 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:43:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:43:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:48.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:48.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:49.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:49.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:49.566 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:43:49.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:43:49.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:49.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:49.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:50.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:50.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:50.808 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:43:50.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:43:50.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:50.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:51.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:52.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:52.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:52.048 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:43:52.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:43:52.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:52.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:52.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:53.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:53.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:53.370 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:43:53.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:43:53.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:53.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:53.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:54.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:54.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:54.614 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:43:54.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:43:54.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:54.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:54.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:55.858 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:43:55.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:43:55.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:55.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:56.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:57.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:57.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:57.497 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:43:57.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:43:57.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:57.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:58.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:43:59.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:43:59.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:43:59.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:43:59.203 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:43:59.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:43:59.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:43:59.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:00.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:00.851 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:44:00.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:44:00.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:00.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:44:01.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:02.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:02.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:02.288 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:44:02.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:44:02.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:02.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:44:02.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:03.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:03.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:03.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:44:03.646 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:44:03.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:03.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:44:04.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:05.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:05.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:05.327 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:44:05.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:44:05.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:05.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:44:05.584 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3712832310,v1:127.0.0.1:6811/3712832310] [v2:127.0.0.1:6812/3712832310,v1:127.0.0.1:6813/3712832310] exists,up b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:44:05.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:44:05.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:44:05.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:44:05.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:44:05.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:44:05.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:44:05.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:44:05.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:44:05.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:44:05.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:44:05.598 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:44:05.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e61fda2e-590f-40dc-8efa-911dab9b31cc' 2024-06-15T10:44:05.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:44:05.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1cG1mxW/zJBAA5elFVCoJsUV0s93Rnh1HbQ== 2024-06-15T10:44:05.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1cG1mxW/zJBAA5elFVCoJsUV0s93Rnh1HbQ=="}' 2024-06-15T10:44:05.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e61fda2e-590f-40dc-8efa-911dab9b31cc -i td/osd-scrub-repair/2/new.json 2024-06-15T10:44:05.985 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:44:06.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:44:06.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1cG1mxW/zJBAA5elFVCoJsUV0s93Rnh1HbQ== --osd-uuid e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:44:06.042 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.039+0000 7fea8ad5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:06.087 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.087+0000 7fea8ad5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:06.089 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.087+0000 7fea8ad5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:06.090 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.087+0000 7fea8ad5b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:44:06.090 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.087+0000 7fea8ad5b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:44:06.090 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.087+0000 7fea8ad5b7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:44:06.170 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:06.167+0000 7fea8ad5b7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:44:12.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:44:12.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:44:12.016 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:44:12.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:44:12.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:44:12.292 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:44:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:44:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:44:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:44:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:44:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:44:12.326 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:12.323+0000 7f79f64917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:12.328 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:12.327+0000 7f79f64917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:12.329 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:12.327+0000 7f79f64917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:44:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:44:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:44:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:44:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:44:12.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:44:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:12.553 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:44:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:44:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:12.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:12.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:12.886 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:12.883+0000 7f79f64917c0 -1 Falling back to public interface 2024-06-15T10:44:13.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:13.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:13.809 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:44:13.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:44:13.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:13.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:14.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:15.050 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:44:15.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:44:15.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:15.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:15.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:15.540 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:44:15.539+0000 7f79f64917c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:44:16.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:16.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:16.304 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:44:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:44:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:16.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:17.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:17.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:17.687 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:44:17.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:44:17.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:17.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:17.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:18.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:18.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:18.929 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:44:18.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:44:18.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:18.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:19.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:20.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:20.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:20.250 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:44:20.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:44:20.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:20.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:20.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:44:21.496 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:44:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:21.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:22.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:22.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:22.739 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:44:22.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:44:22.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:22.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:22.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:23.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:23.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:23.984 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:44:23.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:44:23.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:23.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:24.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:25.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:25.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:25.424 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:44:25.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:44:25.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:25.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:25.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:26.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:26.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:26.669 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:44:26.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:44:26.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:26.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:26.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:27.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:27.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:27.911 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:44:27.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:44:27.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:27.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:28.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:29.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:29.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:29.563 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:44:29.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:44:29.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:29.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:30.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:31.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:31.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:31.267 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:44:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:44:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:31.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:31.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:32.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:32.784 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:44:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:44:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:32.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:33.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:34.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:34.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:34.229 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:44:34.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:44:34.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:34.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:34.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:35.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:35.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:35.692 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:44:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:44:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:35.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:36.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:37.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:37.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:37.022 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:44:37.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:44:37.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:37.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:37.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:44:38.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:44:38.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:44:38.476 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:44:38.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:44:38.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:44:38.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:44:38.730 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/443732415,v1:127.0.0.1:6819/443732415] [v2:127.0.0.1:6820/443732415,v1:127.0.0.1:6821/443732415] exists,up e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:44:38.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:44:38.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:44:38.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:44:38.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-15T10:44:38.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:44:38.962 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:44:38.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:44:38.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:44:39.458 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:44:39.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:44:40.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:44:41.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-15T10:44:41.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-15T10:44:42.301 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' created 2024-06-15T10:44:42.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:44:43.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-06-15T10:44:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:44:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:44:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T10:44:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:44:43.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-15T10:44:43.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:44:43.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:44:44.411 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:44:44.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:44:45.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T10:44:45.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:44:45.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:44:45.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:44:45.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:44:45.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:44:45.438 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:44:45.439 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:44:45.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:44:45.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:44:45.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:44:45.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:44:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:44:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:44:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:44:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:44:45.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:45.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:44:45.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-15T10:44:45.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-15T10:44:45.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-15T10:44:45.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:45.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:44:46.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-15T10:44:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-15T10:44:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-06-15T10:44:46.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:46.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:44:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-06-15T10:44:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-06-15T10:44:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509447' 2024-06-15T10:44:46.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:46.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:46.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-15T10:44:46.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:44:46.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-15T10:44:46.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:46.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-15T10:44:46.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-15T10:44:46.306 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836499 2024-06-15T10:44:46.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:44:46.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-15T10:44:46.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:46.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-15T10:44:46.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:46.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:44:46.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-15T10:44:46.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-15T10:44:46.551 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672973 2024-06-15T10:44:46.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-15T10:44:46.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:44:46.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-15T10:44:46.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:46.792 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-06-15T10:44:46.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:46.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:44:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-06-15T10:44:46.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:46.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-06-15T10:44:46.797 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509447 2024-06-15T10:44:46.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-06-15T10:44:46.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:44:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-06-15T10:44:47.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:44:48.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:44:48.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:44:48.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-06-15T10:44:48.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:44:48.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:48.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:48.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:44:48.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:48.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:48.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:48.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:48.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:48.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:48.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:48.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:44:48.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:48.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:48.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:49.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:44:49.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-15T10:44:49.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:44:49.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-15T10:44:49.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:44:49.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:44:49.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:44:49.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:49.196 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:49.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:44:49.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:49.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:49.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:44:49.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:44:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:44:49.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:44:49.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:44:49.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:44:49.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:44:49.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:44:49.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:44:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:44:50.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:50.151 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:50.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:44:50.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:50.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:50.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:50.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:44:50.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:44:50.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:44:50.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:44:50.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:44:50.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:44:50.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:44:50.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:44:50.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:44:50.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:44:50.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:44:50.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:44:50.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:44:50.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:44:51.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:44:51.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:51.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:51.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:51.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:51.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:51.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:51.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:51.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:44:51.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:51.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:51.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:51.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:44:51.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:44:51.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:44:51.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:44:51.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:44:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:44:51.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:44:51.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:44:51.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:44:52.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:44:52.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:44:52.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-06-15T10:44:52.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:44:52.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-06-15T10:44:52.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:44:52.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:52.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:52.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:52.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:52.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:52.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:52.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:53.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:44:53.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:53.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:53.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:53.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:44:53.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:44:53.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:44:53.455 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:44:53.455 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:44:53.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:44:53.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:44:53.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:44:53.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:53.805 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:44:53.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-06-15T10:44:53.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-06-15T10:44:53.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-06-15T10:44:53.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:53.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:44:54.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-15T10:44:54.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-15T10:44:54.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-06-15T10:44:54.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:44:54.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:44:54.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-06-15T10:44:54.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-06-15T10:44:54.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509450' 2024-06-15T10:44:54.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:54.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:54.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-06-15T10:44:54.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:44:54.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-06-15T10:44:54.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:54.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-06-15T10:44:54.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-15T10:44:54.285 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836502 2024-06-15T10:44:54.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:44:54.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-06-15T10:44:54.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:54.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-15T10:44:54.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:54.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:44:54.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-15T10:44:54.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:54.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-15T10:44:54.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-15T10:44:54.516 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2024-06-15T10:44:54.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:44:54.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-06-15T10:44:54.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:44:54.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-06-15T10:44:54.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:44:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:44:54.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-06-15T10:44:54.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:44:54.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-06-15T10:44:54.756 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509450 2024-06-15T10:44:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-15T10:44:54.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:44:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-06-15T10:44:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:44:56.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:44:56.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:44:56.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-06-15T10:44:56.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:44:57.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:44:57.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:44:57.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-15T10:44:57.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:44:57.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:57.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:57.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:44:57.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:44:57.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:44:57.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:44:57.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:44:57.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:44:57.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:44:58.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:44:58.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:44:58.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:44:58.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:44:58.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:44:58.588 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:44:58.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:44:59.166 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:44:59.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:44:59.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:44:59.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:44:59.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-15T10:44:59.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:44:59.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:44:59.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:44:59.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:44:59.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:44:59.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:44:59.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:44:59.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:00.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:45:00.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:00.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:45:00.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:02.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:02.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:02.449 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:02.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:45:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:45:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:45:02.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:02.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:45:02.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:45:02.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:02.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:02.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:02.486 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:02.486+0000 7f5a379337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:02.488 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:02.490+0000 7f5a379337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:02.489 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:02.490+0000 7f5a379337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:45:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:45:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:02.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:02.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:02.694 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:02.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:02.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:02.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:03.030 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:03.030+0000 7f5a379337c0 -1 Falling back to public interface 2024-06-15T10:45:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:45:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:03.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:03.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:04.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:05.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:05.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:05.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:45:05.176 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:45:05.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:05.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:05.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:05.680 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:05.682+0000 7f5a379337c0 -1 osd.1 34 log_to_monitors true 2024-06-15T10:45:06.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:06.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:06.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:45:06.424 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:45:06.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:06.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 39 up_thru 29 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/3161750877,v1:127.0.0.1:6811/3161750877] [v2:127.0.0.1:6812/3161750877,v1:127.0.0.1:6813/3161750877] exists,up b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:45:06.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:45:06.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:45:06.718 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:45:06.718 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:45:06.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:45:06.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:45:06.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:45:06.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:45:06.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:45:06.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:45:06.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:45:06.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:45:06.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:45:07.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:45:07.181 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:07.182 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:45:07.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:45:07.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:07.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:45:07.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-06-15T10:45:07.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-06-15T10:45:07.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-06-15T10:45:07.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:07.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:45:07.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-06-15T10:45:07.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-06-15T10:45:07.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-167503724546' 2024-06-15T10:45:07.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:07.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:45:07.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-15T10:45:07.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-15T10:45:07.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-167503724546 2-64424509453' 2024-06-15T10:45:07.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:07.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-06-15T10:45:07.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:07.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:45:07.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-06-15T10:45:07.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:07.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-06-15T10:45:07.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-06-15T10:45:07.780 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836505 2024-06-15T10:45:07.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:07.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-06-15T10:45:07.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:45:09.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:45:09.002 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:09.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-06-15T10:45:09.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:09.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:09.243 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-06-15T10:45:09.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:45:09.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-06-15T10:45:09.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:09.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-06-15T10:45:09.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-06-15T10:45:09.246 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 167503724546 2024-06-15T10:45:09.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:45:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-06-15T10:45:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:09.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-15T10:45:09.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:09.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:45:09.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-15T10:45:09.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-15T10:45:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-15T10:45:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509453 2024-06-15T10:45:09.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:45:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-06-15T10:45:09.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:45:09.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:09.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:09.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:45:09.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:45:09.988 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:45:10.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:45:10.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:45:10.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:10.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:10.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:45:10.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:45:10.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:45:10.820 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:45:10.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:45:11.171 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:45:11.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:45:11.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:45:11.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:45:11.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-15T10:45:11.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:45:11.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:45:11.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:1+0 records in 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:1+0 records out 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000143518 s, 14.3 MB/s 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:45:11.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:45:11.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:45:11.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:45:11.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:45:11.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:12.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:45:12.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:45:14.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:45:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:14.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:14.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:14.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:14.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:14.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:45:14.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:45:14.380 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:45:14.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:14.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:45:14.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:45:14.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:14.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:14.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:14.414 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:14.414+0000 7f0f41e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:14.416 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:14.418+0000 7f0f41e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:14.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:14.418+0000 7f0f41e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:14.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:14.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:14.623 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:14.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:14.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:14.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:14.969 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:14.970+0000 7f0f41e657c0 -1 Falling back to public interface 2024-06-15T10:45:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:45:15.890 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:15.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:15.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:16.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:17.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:17.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:17.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:45:17.131 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:45:17.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:17.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:17.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:17.624 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:17.626+0000 7f0f41e657c0 -1 osd.0 43 log_to_monitors true 2024-06-15T10:45:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:45:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:45:18.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:18.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/1801567691,v1:127.0.0.1:6803/1801567691] [v2:127.0.0.1:6804/1801567691,v1:127.0.0.1:6805/1801567691] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:45:18.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:45:18.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:45:18.799 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:45:18.799 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:45:18.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:45:18.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:45:18.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:45:18.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:45:18.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:45:18.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:45:18.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:45:18.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:45:18.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:45:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:45:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:45:19.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:45:19.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:19.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:45:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-06-15T10:45:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-06-15T10:45:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-06-15T10:45:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:19.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:45:19.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-06-15T10:45:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-06-15T10:45:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724549' 2024-06-15T10:45:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:19.510 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:45:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-06-15T10:45:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-06-15T10:45:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724549 2-64424509457' 2024-06-15T10:45:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:19.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-06-15T10:45:19.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:19.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:45:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-06-15T10:45:19.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:19.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-06-15T10:45:19.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-06-15T10:45:19.657 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 206158430210 2024-06-15T10:45:19.658 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:19.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-06-15T10:45:19.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:45:20.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:45:20.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:21.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-06-15T10:45:21.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:21.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724549 2024-06-15T10:45:21.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:21.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:45:21.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724549 2024-06-15T10:45:21.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:21.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-06-15T10:45:21.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-06-15T10:45:21.139 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 167503724549 2024-06-15T10:45:21.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:45:21.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724549 -lt 167503724549 2024-06-15T10:45:21.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:21.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-06-15T10:45:21.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:21.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:45:21.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-06-15T10:45:21.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:21.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-06-15T10:45:21.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-15T10:45:21.386 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509457 2024-06-15T10:45:21.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:45:21.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-15T10:45:21.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:45:21.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:21.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:21.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:45:21.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:45:21.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:45:21.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:45:21.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:45:21.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:45:21.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:45:21.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:45:22.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:45:22.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:45:22.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:22.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:45:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:45:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:45:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:45:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:45:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:45:22.742 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:45:22.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:45:23.278 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:45:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:45:23.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:45:23.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:45:23.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-15T10:45:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:45:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:45:23.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:45:23.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:45:23.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:45:23.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:45:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:45:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:45:23.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:23.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:45:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-15T10:45:25.408 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-15T10:45:25.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:45:25.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:25.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:25.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:25.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:25.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:25.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:25.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:25.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:25.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:25.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:45:25.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:45:25.946 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:25.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:25.983 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:25.986+0000 7f29bf56c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:25.986 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:25.986+0000 7f29bf56c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:25.987 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:25.990+0000 7f29bf56c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:26.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:26.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:26.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:26.530 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:26.530+0000 7f29bf56c7c0 -1 Falling back to public interface 2024-06-15T10:45:27.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:27.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:27.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:45:27.424 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:27.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:27.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:27.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:28.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:28.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:29.082 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:29.082+0000 7f29bf56c7c0 -1 osd.1 52 log_to_monitors true 2024-06-15T10:45:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:45:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:45:29.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:29.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/3192440233,v1:127.0.0.1:6811/3192440233] [v2:127.0.0.1:6812/3192440233,v1:127.0.0.1:6813/3192440233] exists,up b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:45:30.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:45:30.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:45:30.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:45:30.411 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:45:30.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:45:30.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:45:30.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:45:30.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:45:30.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:45:30.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:45:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:45:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:45:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:45:30.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:30.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:45:31.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-06-15T10:45:31.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-06-15T10:45:31.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213' 2024-06-15T10:45:31.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:31.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:45:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-06-15T10:45:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-06-15T10:45:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874' 2024-06-15T10:45:31.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:31.304 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:45:31.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-06-15T10:45:31.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-06-15T10:45:31.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874 2-64424509460' 2024-06-15T10:45:31.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:31.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430213 2024-06-15T10:45:31.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:31.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:45:31.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430213 2024-06-15T10:45:31.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:31.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-06-15T10:45:31.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430213' 2024-06-15T10:45:31.457 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 206158430213 2024-06-15T10:45:31.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:31.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-06-15T10:45:31.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:45:32.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:45:32.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:32.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430213 2024-06-15T10:45:32.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:32.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-06-15T10:45:32.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:32.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:45:32.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-06-15T10:45:32.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:32.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-06-15T10:45:32.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-06-15T10:45:32.933 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 244813135874 2024-06-15T10:45:32.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:45:33.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-06-15T10:45:33.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:33.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-06-15T10:45:33.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:33.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:45:33.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-06-15T10:45:33.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:33.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-06-15T10:45:33.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-06-15T10:45:33.174 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509460 2024-06-15T10:45:33.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:45:33.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-06-15T10:45:33.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:45:33.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:33.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:33.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:45:33.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:45:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:45:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:45:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:45:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:45:33.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:45:33.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:45:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:45:33.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:45:33.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:33.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:34.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:45:34.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:45:34.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:45:34.604 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:45:34.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:45:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:45:35.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:45:35.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:45:35.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:45:35.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-15T10:45:35.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:45:35.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:45:35.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-15T10:45:35.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-15T10:45:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-15T10:45:35.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:45:35.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:45:35.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:45:35.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:45:35.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:45:35.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:45:35.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:36.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:45:36.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:45:38.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:45:38.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:38.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:38.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:38.158 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:38.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:38.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:45:38.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:38.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:45:38.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:45:38.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:38.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:38.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:38.195 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:38.194+0000 7fd2cd9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:38.198 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:38.198+0000 7fd2cd9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:38.199 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:38.202+0000 7fd2cd9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:38.400 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:38.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:38.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:38.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:38.734 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:38.734+0000 7fd2cd9607c0 -1 Falling back to public interface 2024-06-15T10:45:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:45:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:39.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:40.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:40.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:40.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:45:40.897 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:45:40.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:40.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:41.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:41.346 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:41.346+0000 7fd2cd9607c0 -1 osd.0 61 log_to_monitors true 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:42.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:45:42.487 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/1472187916,v1:127.0.0.1:6803/1472187916] [v2:127.0.0.1:6804/1472187916,v1:127.0.0.1:6805/1472187916] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:45:42.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:45:42.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:45:42.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:45:42.489 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:45:42.489 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:45:42.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:45:42.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:45:42.489 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:45:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:45:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:45:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:45:42.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:45:42.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:45:42.603 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:45:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:45:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:42.905 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:45:42.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:45:42.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:42.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:45:43.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-06-15T10:45:43.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-06-15T10:45:43.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-06-15T10:45:43.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:43.126 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:45:43.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135877 2024-06-15T10:45:43.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135877 2024-06-15T10:45:43.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877' 2024-06-15T10:45:43.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:43.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:45:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-15T10:45:43.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-15T10:45:43.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877 2-64424509464' 2024-06-15T10:45:43.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:43.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-06-15T10:45:43.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:43.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:45:43.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-06-15T10:45:43.447 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:43.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-06-15T10:45:43.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-06-15T10:45:43.448 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 283467841538 2024-06-15T10:45:43.448 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:43.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-06-15T10:45:43.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:45:44.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:45:44.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:44.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-06-15T10:45:44.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:44.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135877 2024-06-15T10:45:44.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:44.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:45:44.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135877 2024-06-15T10:45:44.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:44.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135877 2024-06-15T10:45:44.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135877' 2024-06-15T10:45:44.928 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 244813135877 2024-06-15T10:45:44.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:45:45.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135877 -lt 244813135877 2024-06-15T10:45:45.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:45.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-15T10:45:45.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:45.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:45:45.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-15T10:45:45.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:45.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-15T10:45:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-15T10:45:45.177 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509464 2024-06-15T10:45:45.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:45:45.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-15T10:45:45.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:45:45.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:45.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:45.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:45:45.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:45:45.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:45:45.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:45:45.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:45:45.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:45:45.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:45:45.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:45:45.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:45:45.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:45:45.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:45.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:45:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:45:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:45:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:45:46.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:45:46.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:45:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:45:46.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:46.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:45:46.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:45:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:45:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:45:48.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:48.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:48.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:48.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:45:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:45:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T10:45:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:48.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:45:48.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:45:48.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:48.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:48.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:48.604 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:48.606+0000 7f19dec127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:48.607 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:48.606+0000 7f19dec127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:48.608 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:48.610+0000 7f19dec127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:48.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:45:48.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:45:48.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:49.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:49.145 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:49.146+0000 7f19dec127c0 -1 Falling back to public interface 2024-06-15T10:45:50.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:50.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:50.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:45:50.062 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:50.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:50.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:45:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:51.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:51.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:51.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:45:51.305 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:45:51.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:51.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:45:51.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:51.755 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:51.754+0000 7f19dec127c0 -1 osd.2 68 log_to_monitors true 2024-06-15T10:45:52.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:45:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:45:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:45:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:52.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:45:52.793 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/537120059,v1:127.0.0.1:6819/537120059] [v2:127.0.0.1:6820/537120059,v1:127.0.0.1:6821/537120059] exists,up e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:45:52.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:45:52.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:45:52.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:45:52.795 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:45:52.795 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:45:52.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:45:52.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:45:52.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:45:52.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:45:52.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:45:52.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:45:52.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:45:52.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:45:52.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:45:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:45:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:45:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:45:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:45:53.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:53.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:45:53.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-06-15T10:45:53.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-06-15T10:45:53.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-06-15T10:45:53.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:53.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:45:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135880 2024-06-15T10:45:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135880 2024-06-15T10:45:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135880' 2024-06-15T10:45:53.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:45:53.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:45:53.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-06-15T10:45:53.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-06-15T10:45:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-244813135880 2-313532612610' 2024-06-15T10:45:53.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:53.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-06-15T10:45:53.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:53.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:45:53.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-06-15T10:45:53.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-06-15T10:45:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-06-15T10:45:53.713 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 283467841541 2024-06-15T10:45:53.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:53.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-06-15T10:45:53.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:45:54.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:45:54.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:45:55.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-06-15T10:45:55.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:55.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135880 2024-06-15T10:45:55.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:55.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:45:55.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:55.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135880 2024-06-15T10:45:55.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135880 2024-06-15T10:45:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135880' 2024-06-15T10:45:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 244813135880 2024-06-15T10:45:55.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:45:55.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135880 -lt 244813135880 2024-06-15T10:45:55.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:45:55.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:45:55.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-06-15T10:45:55.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:45:55.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:45:55.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-06-15T10:45:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-06-15T10:45:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-06-15T10:45:55.457 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 313532612610 2024-06-15T10:45:55.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:45:55.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-06-15T10:45:55.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:45:55.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:55.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:55.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:45:55.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:45:55.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:45:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:45:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:45:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:45:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:45:55.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:45:56.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:45:56.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:45:56.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:45:56.193 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:45:56.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:45:56.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:45:56.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:45:56.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:45:56.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:45:56.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:45:56.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:45:58.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:45:58.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:45:58.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:45:58.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:45:58.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:45:58.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:45:58.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:45:58.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:45:58.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:45:58.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:45:58.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:45:58.758 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T10:45:58.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:45:58.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:45:58.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:45:58.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:45:58.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:45:58.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:45:58.790 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:58.790+0000 7f2fb41f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:58.793 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:58.794+0000 7f2fb41f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:58.795 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:58.794+0000 7f2fb41f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:45:59.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:45:59.027 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:45:59.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:45:59.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:45:59.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:45:59.354 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:45:59.354+0000 7f2fb41f97c0 -1 Falling back to public interface 2024-06-15T10:46:00.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:00.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:00.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:00.284 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:00.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:00.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:46:00.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:01.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:01.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:01.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:46:01.529 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:46:01.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:01.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:46:01.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:01.775 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:01.774+0000 7f2fb41f97c0 -1 osd.2 75 log_to_monitors true 2024-06-15T10:46:02.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:02.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:02.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:46:02.771 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:46:02.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:02.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:46:03.135 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/2919679686,v1:127.0.0.1:6819/2919679686] [v2:127.0.0.1:6820/2919679686,v1:127.0.0.1:6821/2919679686] exists,up e61fda2e-590f-40dc-8efa-911dab9b31cc 2024-06-15T10:46:03.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:46:03.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:46:03.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:46:03.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:46:03.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:46:03.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:46:03.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:46:03.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:46:03.138 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:46:03.138 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:46:03.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:46:03.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:46:03.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:46:03.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:46:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:46:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:46:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:46:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:46:03.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:03.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:46:03.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841544 2024-06-15T10:46:03.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841544 2024-06-15T10:46:03.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544' 2024-06-15T10:46:03.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:03.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:46:03.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135883 2024-06-15T10:46:03.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135883 2024-06-15T10:46:03.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135883' 2024-06-15T10:46:03.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:03.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:46:03.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-06-15T10:46:03.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-06-15T10:46:03.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-244813135883 2-343597383682' 2024-06-15T10:46:03.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:03.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841544 2024-06-15T10:46:03.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:03.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:46:03.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841544 2024-06-15T10:46:03.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841544 2024-06-15T10:46:03.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841544' 2024-06-15T10:46:03.941 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 283467841544 2024-06-15T10:46:03.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:04.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841544 2024-06-15T10:46:04.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:05.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:46:05.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:05.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841544 -lt 283467841544 2024-06-15T10:46:05.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:05.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135883 2024-06-15T10:46:05.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:05.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:46:05.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135883 2024-06-15T10:46:05.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:05.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135883 2024-06-15T10:46:05.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135883' 2024-06-15T10:46:05.405 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 244813135883 2024-06-15T10:46:05.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:46:05.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135883 -lt 244813135883 2024-06-15T10:46:05.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:05.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-06-15T10:46:05.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:05.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:46:05.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-06-15T10:46:05.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:05.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-06-15T10:46:05.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-06-15T10:46:05.655 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383682 2024-06-15T10:46:05.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:05.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-06-15T10:46:05.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:46:05.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:05.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:06.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:46:06.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:06.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:06.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:06.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:06.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:06.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:06.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:06.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:46:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:06.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:06.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:46:06.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:46:06.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:46:06.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-15T10:46:06.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:46:06.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:46:06.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:46:06.989 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:46:07.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:46:07.491 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:46:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:46:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:46:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:46:07.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-15T10:46:07.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:46:07.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:46:07.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:2+0 records in 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:2+0 records out 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000181576 s, 22.6 MB/s 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:46:07.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:46:07.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:46:07.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:46:07.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:46:07.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:46:08.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:46:08.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:46:08.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:46:08.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:46:08.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:46:08.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:46:08.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:46:08.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:46:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:46:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:46:10.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:46:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:46:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:46:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:46:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:46:10.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:46:10.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:46:10.207 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:10.205+0000 7faad319a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:10.210 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:10.209+0000 7faad319a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:10.211 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:10.213+0000 7faad319a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:10.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:46:10.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:46:10.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:10.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:10.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:10.746 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:10.745+0000 7faad319a7c0 -1 Falling back to public interface 2024-06-15T10:46:11.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:11.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:11.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:11.671 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:11.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:11.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:11.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:13.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:13.424 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:13.425+0000 7faad319a7c0 -1 osd.1 84 log_to_monitors true 2024-06-15T10:46:14.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:14.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:14.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:46:14.179 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:46:14.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:14.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:14.474 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/1057594261,v1:127.0.0.1:6811/1057594261] [v2:127.0.0.1:6812/1057594261,v1:127.0.0.1:6813/1057594261] exists,up b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:46:14.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:46:14.476 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:46:14.476 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:46:14.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:46:14.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:46:14.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:46:14.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:46:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:46:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:46:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:46:14.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:46:14.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:46:14.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:46:14.884 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:14.884 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:46:14.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:46:14.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:14.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:46:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841547 2024-06-15T10:46:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841547 2024-06-15T10:46:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547' 2024-06-15T10:46:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:15.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:46:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-06-15T10:46:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-06-15T10:46:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-382252089346' 2024-06-15T10:46:15.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:15.199 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:46:15.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383685 2024-06-15T10:46:15.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383685 2024-06-15T10:46:15.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-382252089346 2-343597383685' 2024-06-15T10:46:15.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:15.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841547 2024-06-15T10:46:15.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:15.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:46:15.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841547 2024-06-15T10:46:15.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:15.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841547 2024-06-15T10:46:15.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841547' 2024-06-15T10:46:15.344 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 283467841547 2024-06-15T10:46:15.344 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:15.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841546 -lt 283467841547 2024-06-15T10:46:15.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:16.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:46:16.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:16.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841547 2024-06-15T10:46:16.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:16.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-06-15T10:46:16.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:16.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:46:16.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:16.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-06-15T10:46:16.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-06-15T10:46:16.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-06-15T10:46:16.817 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 382252089346 2024-06-15T10:46:16.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:46:17.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-06-15T10:46:17.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:17.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383685 2024-06-15T10:46:17.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:17.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:46:17.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383685 2024-06-15T10:46:17.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:17.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383685 2024-06-15T10:46:17.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383685' 2024-06-15T10:46:17.086 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383685 2024-06-15T10:46:17.087 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:17.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383685 -lt 343597383685 2024-06-15T10:46:17.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:46:17.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:17.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:17.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:46:17.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:17.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:17.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:17.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:17.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:17.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:17.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:17.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:46:17.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:17.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:17.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:46:18.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-15T10:46:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:46:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-15T10:46:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:46:18.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:46:18.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:46:18.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:18.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:18.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:18.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:18.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:18.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:18.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:18.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:46:18.437 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:18.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:18.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:18.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:46:18.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:46:18.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:46:18.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:46:18.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:46:19.135 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:46:19.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:46:19.553 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:46:19.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:46:19.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:46:19.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:46:19.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-15T10:46:19.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:46:19.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:46:19.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:46:19.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:46:19.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:46:19.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:20.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:46:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:46:22.223 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:46:22.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:46:22.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:46:22.224 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:46:22.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:22.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:22.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:46:22.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:46:22.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:46:22.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:46:22.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:46:22.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:46:22.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:46:22.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:46:22.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:46:22.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:46:22.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:46:22.258 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:22.257+0000 7ff8476aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:22.261 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:22.261+0000 7ff8476aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:22.263 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:22.261+0000 7ff8476aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:22.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:22.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:22.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:22.797 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:22.797+0000 7ff8476aa7c0 -1 Falling back to public interface 2024-06-15T10:46:23.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:23.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:23.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:23.698 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:23.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:23.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:23.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:46:24.939 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:46:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:24.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:25.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:25.498 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:25.497+0000 7ff8476aa7c0 -1 osd.0 93 log_to_monitors true 2024-06-15T10:46:26.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:26.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:26.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:46:26.183 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:46:26.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:26.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:26.441 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6802/4141047600,v1:127.0.0.1:6803/4141047600] [v2:127.0.0.1:6804/4141047600,v1:127.0.0.1:6805/4141047600] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:46:26.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:46:26.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:46:26.443 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:46:26.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:26.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:46:27.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-06-15T10:46:27.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-06-15T10:46:27.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-06-15T10:46:27.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:27.116 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:46:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089349 2024-06-15T10:46:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089349 2024-06-15T10:46:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349' 2024-06-15T10:46:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:27.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:46:27.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383689 2024-06-15T10:46:27.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383689 2024-06-15T10:46:27.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349 2-343597383689' 2024-06-15T10:46:27.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:27.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-06-15T10:46:27.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:27.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:46:27.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-06-15T10:46:27.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:27.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-06-15T10:46:27.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-06-15T10:46:27.466 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 420906795010 2024-06-15T10:46:27.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:27.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-06-15T10:46:27.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:46:28.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:28.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-06-15T10:46:28.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:28.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089349 2024-06-15T10:46:28.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:28.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:46:28.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089349 2024-06-15T10:46:28.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:28.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089349 2024-06-15T10:46:28.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089349' 2024-06-15T10:46:28.929 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 382252089349 2024-06-15T10:46:28.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:46:29.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089349 -lt 382252089349 2024-06-15T10:46:29.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:29.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:29.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383689 2024-06-15T10:46:29.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:46:29.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383689 2024-06-15T10:46:29.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:29.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383689 2024-06-15T10:46:29.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383689' 2024-06-15T10:46:29.179 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383689 2024-06-15T10:46:29.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:29.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383689 2024-06-15T10:46:29.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:46:29.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:29.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:29.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:46:29.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:29.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:29.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:29.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:29.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:29.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:29.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:46:29.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:29.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:29.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:30.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-15T10:46:30.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-06-15T10:46:30.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:46:30.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-06-15T10:46:30.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:46:30.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:46:30.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:46:30.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:30.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:30.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:30.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:30.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:30.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:30.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:30.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:46:30.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:30.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:30.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:46:30.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:46:30.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:46:30.800 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:46:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:46:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:46:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:46:31.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:46:31.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:46:31.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:46:31.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:31.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:31.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:31.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:31.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:31.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:31.296 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:31.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:46:31.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:31.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:31.536 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:46:31.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:46:32.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:46:32.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:46:32.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:46:32.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:46:32.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:46:32.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:46:32.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:32.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:32.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:46:32.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:32.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:32.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:32.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:46:32.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:46:32.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:46:32.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-15T10:46:32.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:46:32.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:46:32.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:46:32.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:46:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:46:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:46:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:46:33.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:46:33.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:46:35.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:46:35.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:46:35.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:46:35.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:46:35.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:46:35.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:46:35.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:46:35.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:46:35.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:46:35.223 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:35.221+0000 7f242c1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:35.225 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:35.225+0000 7f242c1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:35.226 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:35.225+0000 7f242c1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:35.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:46:35.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:35.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:35.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:35.770 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:35.769+0000 7f242c1cd7c0 -1 Falling back to public interface 2024-06-15T10:46:36.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:36.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:36.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:36.679 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:36.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:36.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:36.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:37.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:37.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:37.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:46:37.923 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:46:37.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:37.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:38.035 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:38.033+0000 7f242c1cd7c0 -1 osd.1 101 log_to_monitors true 2024-06-15T10:46:38.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:46:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:46:39.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:39.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:46:39.413 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [89,101) [v2:127.0.0.1:6810/3422181691,v1:127.0.0.1:6811/3422181691] [v2:127.0.0.1:6812/3422181691,v1:127.0.0.1:6813/3422181691] exists,up b645dff1-d4a4-407f-a604-f4118837f98f 2024-06-15T10:46:39.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:46:39.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:46:39.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:46:39.415 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:46:39.415 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:46:39.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:46:39.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:46:39.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:46:39.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:46:39.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:46:39.767 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:39.767 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:46:39.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:46:39.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:39.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:46:39.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795013 2024-06-15T10:46:39.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795013 2024-06-15T10:46:39.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013' 2024-06-15T10:46:39.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:39.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:46:40.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-06-15T10:46:40.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-06-15T10:46:40.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378' 2024-06-15T10:46:40.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:40.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:46:40.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383692 2024-06-15T10:46:40.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383692 2024-06-15T10:46:40.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378 2-343597383692' 2024-06-15T10:46:40.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:40.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795013 2024-06-15T10:46:40.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:40.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:46:40.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795013 2024-06-15T10:46:40.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:40.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795013 2024-06-15T10:46:40.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-06-15T10:46:40.232 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 420906795013 2024-06-15T10:46:40.232 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:40.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795013 -lt 420906795013 2024-06-15T10:46:40.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:40.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:40.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-06-15T10:46:40.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:46:40.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-06-15T10:46:40.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:40.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-06-15T10:46:40.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-06-15T10:46:40.469 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 455266533378 2024-06-15T10:46:40.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:46:40.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-06-15T10:46:40.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:40.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383692 2024-06-15T10:46:40.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:40.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:46:40.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383692 2024-06-15T10:46:40.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:40.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383692 2024-06-15T10:46:40.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383692' 2024-06-15T10:46:40.709 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383692 2024-06-15T10:46:40.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:40.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383691 -lt 343597383692 2024-06-15T10:46:40.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:41.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:46:41.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:42.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383691 -lt 343597383692 2024-06-15T10:46:42.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:43.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:46:43.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:43.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383693 -lt 343597383692 2024-06-15T10:46:43.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:46:43.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:43.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:43.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:46:43.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:43.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:43.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:43.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:43.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:43.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:43.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:43.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:46:43.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:43.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:43.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:46:44.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:46:44.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:46:44.590 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:46:44.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:46:44.950 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:46:44.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:46:44.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:46:44.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:46:45.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-15T10:46:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:46:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:46:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-15T10:46:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-15T10:46:45.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:46:45.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:46:45.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:46:45.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:45.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-15T10:46:47.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:46:47.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:46:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:46:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:46:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:46:47.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:46:47.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:46:47.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:46:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:46:47.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:46:47.848 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:46:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:46:47.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:46:47.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:46:47.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:46:47.886 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:47.885+0000 7fb9ce3437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:47.888 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:47.889+0000 7fb9ce3437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:47.890 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:47.889+0000 7fb9ce3437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:48.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:46:48.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:48.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:48.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:48.425 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:48.425+0000 7fb9ce3437c0 -1 Falling back to public interface 2024-06-15T10:46:49.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:49.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:49.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:49.340 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:49.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:49.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:49.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:50.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:50.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:50.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:46:50.581 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:46:50.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:50.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:50.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:50.937 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:50.937+0000 7fb9ce3437c0 -1 osd.0 110 log_to_monitors true 2024-06-15T10:46:51.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:46:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:46:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:51.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [98,110) [v2:127.0.0.1:6802/2209808915,v1:127.0.0.1:6803/2209808915] [v2:127.0.0.1:6804/2209808915,v1:127.0.0.1:6805/2209808915] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:46:52.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:46:52.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:46:52.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:46:52.137 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:46:52.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:46:52.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:46:52.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:46:52.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:46:52.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:46:52.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:46:52.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:46:52.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:46:52.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:46:52.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:46:52.656 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:52.656 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:46:52.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:46:52.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:52.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:46:52.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-06-15T10:46:52.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-06-15T10:46:52.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042' 2024-06-15T10:46:52.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:52.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:46:52.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-06-15T10:46:52.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-06-15T10:46:52.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381' 2024-06-15T10:46:52.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:46:52.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:46:53.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383696 2024-06-15T10:46:53.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383696 2024-06-15T10:46:53.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381 2-343597383696' 2024-06-15T10:46:53.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:53.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-493921239042 2024-06-15T10:46:53.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:53.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:46:53.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-493921239042 2024-06-15T10:46:53.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:53.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-06-15T10:46:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-06-15T10:46:53.096 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 493921239042 2024-06-15T10:46:53.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:53.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-06-15T10:46:53.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:46:54.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:46:54.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:46:54.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239042 -lt 493921239042 2024-06-15T10:46:54.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:54.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533381 2024-06-15T10:46:54.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:54.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:46:54.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533381 2024-06-15T10:46:54.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:54.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-06-15T10:46:54.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533381' 2024-06-15T10:46:54.574 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 455266533381 2024-06-15T10:46:54.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:46:54.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533381 2024-06-15T10:46:54.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:46:54.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383696 2024-06-15T10:46:54.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:46:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:46:54.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383696 2024-06-15T10:46:54.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:46:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383696 2024-06-15T10:46:54.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383696' 2024-06-15T10:46:54.826 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383696 2024-06-15T10:46:54.826 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:46:55.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383696 -lt 343597383696 2024-06-15T10:46:55.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:46:55.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:55.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:55.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:46:55.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:46:55.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:46:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:46:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:46:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:46:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:46:55.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:46:55.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:46:55.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:46:55.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:46:55.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:46:55.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:46:55.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:46:55.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:46:56.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:56.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:46:58.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:46:58.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:46:58.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:46:58.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:46:58.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:46:58.492 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:46:58.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:46:58.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:46:58.494 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:46:58.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:46:58.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:46:58.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:46:58.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:46:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:46:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:46:58.528 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:58.529+0000 7f93bcf207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:58.531 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:58.529+0000 7f93bcf207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:58.532 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:58.533+0000 7f93bcf207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:58.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:46:58.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:46:59.078 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:46:59.077+0000 7f93bcf207c0 -1 Falling back to public interface 2024-06-15T10:46:59.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:46:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:46:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:46:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:46:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:46:59.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:00.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:01.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:01.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:01.559 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:01.561+0000 7f93bcf207c0 -1 osd.0 117 log_to_monitors true 2024-06-15T10:47:02.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:02.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:02.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:47:02.473 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:47:02.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:02.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:02.716 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/3821670100,v1:127.0.0.1:6803/3821670100] [v2:127.0.0.1:6804/3821670100,v1:127.0.0.1:6805/3821670100] exists,up bcbf8693-030a-4230-97ca-85019753e5d6 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:47:02.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:47:02.718 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:47:02.718 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:47:02.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:47:02.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:47:02.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:47:02.837 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:47:03.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:47:03.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-06-15T10:47:03.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-06-15T10:47:03.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-06-15T10:47:03.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:47:03.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:47:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-06-15T10:47:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-06-15T10:47:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533385' 2024-06-15T10:47:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:47:03.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:47:03.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383699 2024-06-15T10:47:03.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383699 2024-06-15T10:47:03.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533385 2-343597383699' 2024-06-15T10:47:03.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:47:03.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-06-15T10:47:03.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:47:03.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:47:03.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-06-15T10:47:03.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:47:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-06-15T10:47:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-06-15T10:47:03.749 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 523986010114 2024-06-15T10:47:03.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:47:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-06-15T10:47:03.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:47:04.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:47:04.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:47:05.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-06-15T10:47:05.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:47:05.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533385 2024-06-15T10:47:05.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:47:05.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:47:05.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533385 2024-06-15T10:47:05.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:47:05.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-06-15T10:47:05.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-06-15T10:47:05.231 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 455266533385 2024-06-15T10:47:05.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:47:05.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-06-15T10:47:05.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:47:05.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383699 2024-06-15T10:47:05.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:47:05.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:47:05.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:47:05.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383699 2024-06-15T10:47:05.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383699 2024-06-15T10:47:05.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383699' 2024-06-15T10:47:05.483 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 343597383699 2024-06-15T10:47:05.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:47:05.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383699 -lt 343597383699 2024-06-15T10:47:05.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:47:05.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:47:05.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:47:06.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:47:06.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:47:06.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:47:06.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:47:06.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:47:06.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:47:06.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:47:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:47:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:47:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:47:06.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-15T10:47:06.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-15T10:47:06.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:47:06.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-15T10:47:06.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-15T10:47:06.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:47:06.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-15T10:47:06.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-15T10:47:06.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-15T10:47:06.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-15T10:47:06.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:06.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:47:06.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:06.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:47:06.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:06.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-15T10:47:07.212 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:47:07.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:07.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:47:07.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:07.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:47:07.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:07.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:47:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:44:44.364020+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:07.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:47:08.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:47:08.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:08.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:47:08.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:08.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:47:08.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:08.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:47:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:44:44.364020+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:08.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:47:09.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:47:09.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:09.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:47:09.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:09.682 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:47:09.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:09.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:47:09.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:44:44.364020+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:09.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:47:10.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:47:10.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:10.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:47:10.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:10.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:47:10.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:10.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:47:11.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:47:07.261559+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:11.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:47:11.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-15T10:47:11.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:47:11.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-15T10:47:11.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:47:11.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-15T10:47:11.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-15T10:47:11.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:47:11.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:47:11.368 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:47:11.369 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:47:11.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:47:11.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:47:11.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:47:11.410 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:47:11.410 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:47:11.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:47:11.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:47:11.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:47:11.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-15T10:47:11.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-15T10:47:11.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-15T10:47:11.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-15T10:47:11.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:47:11.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:11.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:47:11.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:11.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:47:11.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:11.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-15T10:47:11.898 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-15T10:44:44.364020+0000 last_deep_scrub_stamp 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:47:11.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:11.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:47:11.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:11.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:47:11.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:11.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:47:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:44:44.364020+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:12.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:47:13.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:47:13.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:13.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:47:13.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:13.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:47:13.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:13.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:47:13.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:44:44.364020+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:13.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:47:14.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:47:14.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:47:14.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:47:14.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:47:14.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:47:14.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:47:14.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:47:14.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:47:12.362114+0000 '>' 2024-06-15T10:44:44.364020+0000 2024-06-15T10:47:14.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:47:14.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-15T10:47:14.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:47:14.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-15T10:47:14.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:47:14.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-15T10:47:14.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-15T10:47:14.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:47:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-06-15T10:47:14.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:47:14.811 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:47:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:47:14.856 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:47:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:47:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:47:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:47:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:47:14.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:47:14.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:47:14.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:47:14.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-15T10:47:14.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-15T10:47:14.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-15T10:47:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' removed 2024-06-15T10:47:15.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:47:15.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:47:15.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:47:15.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:47:15.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:47:15.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:47:15.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:47:15.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:47:15.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:47:15.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:47:15.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:47:15.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:47:15.633 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:47:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:47:15.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:47:15.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:47:15.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:47:15.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:47:15.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:47:15.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:47:15.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:47:15.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:47:15.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:47:15.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:47:15.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:15.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:15.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:47:15.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:47:15.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:47:15.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:47:15.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:47:15.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:47:15.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:47:15.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:47:15.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:47:15.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:47:15.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:47:15.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:47:15.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:47:15.685 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:47:15.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:47:15.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:47:15.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:47:15.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:47:15.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:47:15.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:47:15.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:47:15.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:47:15.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:47:15.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:47:15.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:47:15.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:15.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:15.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:47:15.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:47:15.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:47:15.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:47:15.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:47:15.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:15.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:15.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:47:15.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-06-15T10:47:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:47:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T10:47:16.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:47:16.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:47:16.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:47:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:47:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:16.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:16.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T10:47:16.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:47:16.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:47:16.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:47:16.288 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:47:16.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:47:16.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:47:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:47:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:47:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:47:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:47:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:47:16.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:47:16.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:47:16.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:47:16.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:47:16.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:47:16.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:47:16.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:47:16.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:47:16.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:47:16.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:47:16.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:47:16.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:47:17.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:47:17.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:47:17.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:47:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:47:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:17.400 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:17.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:17.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:47:17.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:47:17.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-06-15T10:47:17.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:47:17.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:17.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:47:17.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:47:17.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:47:17.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:47:17.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:47:17.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:47:17.445 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:47:17.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b38592c-8df4-4003-ba9e-814328cbd2f2' 2024-06-15T10:47:17.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:47:17.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1cW1mdQLIGxAAhIPhxPOt0k4XqZ/nh4ewEw== 2024-06-15T10:47:17.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1cW1mdQLIGxAAhIPhxPOt0k4XqZ/nh4ewEw=="}' 2024-06-15T10:47:17.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b38592c-8df4-4003-ba9e-814328cbd2f2 -i td/osd-scrub-repair/0/new.json 2024-06-15T10:47:17.700 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:47:17.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:47:17.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1cW1mdQLIGxAAhIPhxPOt0k4XqZ/nh4ewEw== --osd-uuid 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:47:17.752 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.752+0000 7f45b3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:17.754 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.752+0000 7f45b3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:17.756 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.756+0000 7f45b3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:17.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.756+0000 7f45b3fc67c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:47:17.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.756+0000 7f45b3fc67c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:47:17.757 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.756+0000 7f45b3fc67c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:47:17.834 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:17.832+0000 7f45b3fc67c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:47:23.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:47:23.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:47:23.494 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:47:23.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:47:23.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:47:23.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:47:23.823 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:47:23.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:47:23.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:47:23.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:47:23.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:47:23.855 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:23.852+0000 7fa0f93547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:23.875 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:23.872+0000 7fa0f93547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:23.876 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:23.876+0000 7fa0f93547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:47:24.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:24.081 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:47:24.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:47:24.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:24.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:24.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:24.438 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:24.436+0000 7fa0f93547c0 -1 Falling back to public interface 2024-06-15T10:47:25.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:25.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:25.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:47:25.314 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:47:25.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:25.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:25.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:26.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:26.555 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:47:26.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:47:26.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:26.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:26.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:26.998 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:26.996+0000 7fa0f93547c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:47:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:27.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:27.798 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:47:27.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:47:27.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:27.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:28.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:29.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:29.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:29.076 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:47:29.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:47:29.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:29.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:29.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:30.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:30.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:30.317 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:47:30.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:47:30.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:30.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:30.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:31.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:31.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:31.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:47:31.641 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:47:31.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:31.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:31.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:32.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:32.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:32.940 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:47:32.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:47:32.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:32.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:33.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:34.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:34.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:34.185 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:47:34.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:47:34.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:34.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:34.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:35.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:35.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:35.434 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:47:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:47:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:35.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:36.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:36.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:36.818 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:47:36.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:47:36.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:36.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:38.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:38.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:38.058 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:47:38.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:47:38.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:38.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:38.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:39.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:39.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:39.299 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:47:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:47:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:39.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:40.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:41.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:47:41.032 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:47:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:41.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:41.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:42.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:42.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:42.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:47:42.351 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:47:42.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:42.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:44.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:44.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:44.223 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:47:44.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:47:44.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:44.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:44.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:45.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:45.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:45.585 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:47:45.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:47:45.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:45.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:46.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:47.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:47.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:47.140 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:47:47.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:47:47.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:47.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:47.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:48.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:48.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:48.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:47:48.848 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:47:48.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:48.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:47:49.087 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3624380128,v1:127.0.0.1:6803/3624380128] [v2:127.0.0.1:6804/3624380128,v1:127.0.0.1:6805/3624380128] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:47:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:47:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:47:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:47:49.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:47:49.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:47:49.091 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:47:49.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:47:49.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:47:49.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:47:49.094 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:47:49.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1c943628-6725-40ce-8d8a-c9d5aec6c772' 2024-06-15T10:47:49.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:47:49.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVcW1mvuUOBxAAgIl3jEGUIcePeitdsRzbJg== 2024-06-15T10:47:49.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVcW1mvuUOBxAAgIl3jEGUIcePeitdsRzbJg=="}' 2024-06-15T10:47:49.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c943628-6725-40ce-8d8a-c9d5aec6c772 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:47:49.399 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:47:49.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:47:49.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVcW1mvuUOBxAAgIl3jEGUIcePeitdsRzbJg== --osd-uuid 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:47:49.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.452+0000 7f047690c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:49.486 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.484+0000 7f047690c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:49.487 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.488+0000 7f047690c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:49.488 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.488+0000 7f047690c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:47:49.488 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.488+0000 7f047690c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:47:49.488 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.488+0000 7f047690c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:47:49.562 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:49.560+0000 7f047690c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:47:55.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:47:55.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:47:55.531 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:47:55.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:47:55.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:47:55.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:47:55.845 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:47:55.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:47:55.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:47:55.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:47:55.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:47:55.878 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:55.876+0000 7fefe06f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:55.897 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:55.896+0000 7fefe06f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:55.896+0000 7fefe06f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:56.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:47:56.079 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:47:56.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:56.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:47:56.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:56.453 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:56.452+0000 7fefe06f97c0 -1 Falling back to public interface 2024-06-15T10:47:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:57.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:57.313 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:47:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:47:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:57.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:47:57.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:58.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:58.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:58.554 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:47:58.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:47:58.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:58.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:47:58.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:47:59.095 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:47:59.096+0000 7fefe06f97c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:47:59.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:47:59.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:47:59.795 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:47:59.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:47:59.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:47:59.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:00.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:01.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:01.129 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:48:01.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:48:01.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:01.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:01.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:02.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:02.369 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:48:02.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:02.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:48:02.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:02.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:02.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:03.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:03.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:03.781 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:48:03.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:48:03.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:03.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:04.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:05.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:05.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:05.090 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:48:05.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:48:05.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:05.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:05.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:06.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:06.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:06.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:48:06.332 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:48:06.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:06.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:06.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:07.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:07.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:07.577 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:48:07.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:48:07.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:07.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:07.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:08.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:08.984 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:48:08.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:48:08.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:08.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:09.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:10.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:10.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:10.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:48:10.229 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:48:10.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:10.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:10.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:11.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:11.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:11.469 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:48:11.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:48:11.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:11.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:11.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:12.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:12.963 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:48:12.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:48:12.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:12.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:13.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:14.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:14.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:14.648 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:48:14.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:48:14.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:14.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:15.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:16.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:16.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:16.303 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:48:16.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:48:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:16.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:16.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:17.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:17.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:17.841 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:48:17.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:48:17.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:17.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:18.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:19.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:19.361 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:48:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:48:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:19.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:19.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:20.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:20.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:20.609 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:48:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:48:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:20.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:21.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:21.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:21.963 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:48:21.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:48:21.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:21.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:48:22.199 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2043846904,v1:127.0.0.1:6811/2043846904] [v2:127.0.0.1:6812/2043846904,v1:127.0.0.1:6813/2043846904] exists,up 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:48:22.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:48:22.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:48:22.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:48:22.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:48:22.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:48:22.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:48:22.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:48:22.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:48:22.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:48:22.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:48:22.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e' 2024-06-15T10:48:22.207 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:48:22.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:48:22.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2cW1ma99hDRAA90E3D59DtXv6PXblDQg7MA== 2024-06-15T10:48:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2cW1ma99hDRAA90E3D59DtXv6PXblDQg7MA=="}' 2024-06-15T10:48:22.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e -i td/osd-scrub-repair/2/new.json 2024-06-15T10:48:22.526 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:48:22.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T10:48:22.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2cW1ma99hDRAA90E3D59DtXv6PXblDQg7MA== --osd-uuid 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:48:22.576 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.575+0000 7f25960ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:22.594 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.591+0000 7f25960ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:22.595 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.595+0000 7f25960ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:22.596 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.595+0000 7f25960ab7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:48:22.596 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.595+0000 7f25960ab7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:48:22.596 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.595+0000 7f25960ab7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T10:48:22.658 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:22.655+0000 7f25960ab7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T10:48:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T10:48:28.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:48:28.490 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T10:48:28.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T10:48:28.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:48:28.760 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T10:48:28.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T10:48:28.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:48:28.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:48:28.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:48:28.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:48:28.792 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:28.791+0000 7fd47e0487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:28.791+0000 7fd47e0487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:28.795+0000 7fd47e0487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:48:29.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T10:48:29.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:48:29.024 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:29.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:29.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:29.353 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:29.351+0000 7fd47e0487c0 -1 Falling back to public interface 2024-06-15T10:48:30.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:30.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:30.273 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:48:30.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:48:30.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:30.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:30.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:31.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:31.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:31.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:48:31.515 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:48:31.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:31.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:31.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:31.959 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:48:31.955+0000 7fd47e0487c0 -1 osd.2 0 log_to_monitors true 2024-06-15T10:48:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:32.756 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:48:32.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:48:32.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:32.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:33.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:34.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:34.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:34.098 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:48:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:48:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:34.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:34.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:35.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:35.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:35.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:48:35.339 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:48:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:35.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:36.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:36.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:36.674 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:48:36.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:48:36.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:36.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:36.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:37.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:37.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:37.924 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:48:37.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:48:37.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:37.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:38.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:39.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:39.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:39.167 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:48:39.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:48:39.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:39.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:39.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:40.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:40.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:48:40.410 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:48:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:40.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:40.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:41.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:41.764 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:48:41.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:48:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:42.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:43.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:43.010 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:48:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:48:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:43.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:43.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:44.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:44.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:44.256 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:48:44.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:48:44.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:44.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:44.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:45.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:45.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:45.501 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:48:45.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:48:45.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:45.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:46.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:47.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:47.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:47.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:48:47.009 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:48:47.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:47.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:47.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:48.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:48.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:48.642 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:48:48.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:48:48.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:48.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:49.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:50.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:50.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:50.159 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:48:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:48:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:50.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:50.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:51.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:51.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:51.687 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:48:51.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:48:51.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:51.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:52.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:53.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:53.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:53.430 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:48:53.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:48:53.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:53.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:54.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:48:55.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:48:55.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:48:55.011 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:48:55.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:48:55.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:48:55.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:48:55.326 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2542193654,v1:127.0.0.1:6819/2542193654] [v2:127.0.0.1:6820/2542193654,v1:127.0.0.1:6821/2542193654] exists,up 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:48:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:48:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:48:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:48:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-06-15T10:48:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:48:55.553 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:48:55.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:48:55.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:48:56.018 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:48:56.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:48:57.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:48:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-06-15T10:48:59.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-15T10:48:59.490 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' created 2024-06-15T10:48:59.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:49:00.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-06-15T10:49:00.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T10:49:00.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T10:49:00.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-15T10:49:00.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T10:49:00.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-06-15T10:49:01.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T10:49:01.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T10:49:01.676 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T10:49:01.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:49:02.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-15T10:49:02.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-15T10:49:03.047 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 3 allow_ec_overwrites to true 2024-06-15T10:49:03.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T10:49:03.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:03.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:03.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:03.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:03.068 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:03.068 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:03.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:03.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:03.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:03.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:03.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:03.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-06-15T10:49:03.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-06-15T10:49:03.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-06-15T10:49:03.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:03.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:03.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-15T10:49:03.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-15T10:49:03.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672973' 2024-06-15T10:49:03.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:03.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:03.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-06-15T10:49:03.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-06-15T10:49:03.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672973 2-64424509448' 2024-06-15T10:49:03.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:03.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-06-15T10:49:03.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:03.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:03.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-06-15T10:49:03.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:03.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-06-15T10:49:03.980 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836500 2024-06-15T10:49:03.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-06-15T10:49:03.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:04.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-06-15T10:49:04.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:05.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:49:05.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:05.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-06-15T10:49:05.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:06.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:49:06.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:06.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836500 2024-06-15T10:49:06.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:06.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-15T10:49:06.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:06.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:49:06.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-15T10:49:06.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:06.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-15T10:49:06.706 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672973 2024-06-15T10:49:06.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-15T10:49:06.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:06.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-15T10:49:06.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:06.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-06-15T10:49:06.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:06.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:49:06.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-06-15T10:49:06.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:06.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-06-15T10:49:06.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-06-15T10:49:06.950 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509448 2024-06-15T10:49:06.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:49:07.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-06-15T10:49:07.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:49:07.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:07.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:07.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:49:07.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:07.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:07.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:07.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:49:07.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:07.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:07.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:07.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:07.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:08.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:08.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:08.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:08.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:08.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:08.047 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:08.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:08.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-06-15T10:49:08.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-06-15T10:49:08.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-06-15T10:49:08.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:08.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:08.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-15T10:49:08.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-15T10:49:08.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-06-15T10:49:08.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:08.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:08.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-06-15T10:49:08.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-06-15T10:49:08.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509450' 2024-06-15T10:49:08.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:08.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-06-15T10:49:08.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:08.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:08.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-06-15T10:49:08.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:08.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-06-15T10:49:08.720 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836502 2024-06-15T10:49:08.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-06-15T10:49:08.720 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:08.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-06-15T10:49:08.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:09.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:49:09.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-06-15T10:49:10.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-15T10:49:10.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:10.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:49:10.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-15T10:49:10.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:10.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-15T10:49:10.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-15T10:49:10.176 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2024-06-15T10:49:10.176 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:10.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-06-15T10:49:10.438 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:10.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-06-15T10:49:10.439 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:10.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:49:10.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-06-15T10:49:10.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:10.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-06-15T10:49:10.443 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509450 2024-06-15T10:49:10.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-15T10:49:10.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:49:10.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-15T10:49:10.686 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:49:10.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:10.687 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:10.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:49:10.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:10.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:10.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:10.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:10.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:10.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:10.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:11.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:49:11.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:11.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:11.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:49:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:49:11.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:49:11.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-06-15T10:49:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:49:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-06-15T10:49:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-06-15T10:49:11.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:49:11.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:49:11.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-06-15T10:49:11.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:49:11.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:49:11.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:49:11.906 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:49:11.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:49:12.249 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:49:12.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:49:12.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:49:12.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:49:12.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-06-15T10:49:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:49:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:49:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-06-15T10:49:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-06-15T10:49:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:12.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:49:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:49:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:49:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:49:12.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:49:12.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:49:12.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:49:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:49:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:49:12.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:49:12.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:49:12.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:12.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:49:12.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:49:12.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:49:12.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:49:12.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:49:12.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:49:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:49:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:49:15.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:49:15.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:49:15.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:49:15.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:49:15.294 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:49:15.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:49:15.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:49:15.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:49:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:49:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:49:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:49:15.328 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:15.330+0000 7f40019aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:15.331 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:15.334+0000 7f40019aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:15.332 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:15.334+0000 7f40019aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:49:15.535 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:49:15.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:15.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:15.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:15.870 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:15.870+0000 7f40019aa7c0 -1 Falling back to public interface 2024-06-15T10:49:16.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:16.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:16.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:49:16.772 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:16.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:16.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:17.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:18.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:18.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:18.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:49:18.017 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:49:18.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:18.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:18.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:18.429 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:18.430+0000 7f40019aa7c0 -1 osd.1 37 log_to_monitors true 2024-06-15T10:49:19.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:19.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:49:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:49:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:19.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/2911862064,v1:127.0.0.1:6811/2911862064] [v2:127.0.0.1:6812/2911862064,v1:127.0.0.1:6813/2911862064] exists,up 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:49:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:49:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:49:19.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:49:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:49:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:19.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:19.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:19.512 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:19.512 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:19.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:19.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:19.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:19.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:19.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:19.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:19.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:19.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:19.630 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:20.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:20.051 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:20.051 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:20.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:20.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:20.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-06-15T10:49:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-06-15T10:49:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-06-15T10:49:20.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:20.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:20.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626434 2024-06-15T10:49:20.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626434 2024-06-15T10:49:20.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-180388626434' 2024-06-15T10:49:20.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:20.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:20.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-06-15T10:49:20.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-06-15T10:49:20.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-180388626434 2-64424509453' 2024-06-15T10:49:20.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:20.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:20.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-06-15T10:49:20.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:20.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-06-15T10:49:20.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:20.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-06-15T10:49:20.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-06-15T10:49:20.491 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836506 2024-06-15T10:49:20.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:20.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-06-15T10:49:20.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:21.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:49:21.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:21.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-06-15T10:49:21.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:21.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:21.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626434 2024-06-15T10:49:21.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:49:21.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626434 2024-06-15T10:49:21.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626434 2024-06-15T10:49:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626434' 2024-06-15T10:49:21.977 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 180388626434 2024-06-15T10:49:21.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:22.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626434 -lt 180388626434 2024-06-15T10:49:22.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:22.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-06-15T10:49:22.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:22.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:49:22.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-06-15T10:49:22.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:22.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-06-15T10:49:22.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-06-15T10:49:22.236 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509453 2024-06-15T10:49:22.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:49:22.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-06-15T10:49:22.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:49:22.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:22.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:22.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:23.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:49:23.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:23.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:23.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:23.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:49:23.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-15T10:49:23.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:49:23.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-15T10:49:23.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:49:23.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:49:23.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:49:23.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:23.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:23.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:23.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:23.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:23.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:23.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:23.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:49:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:23.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:49:23.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:49:23.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:49:24.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:49:24.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:49:24.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:49:24.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:49:24.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:49:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:49:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:24.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:24.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:24.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:24.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:24.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:24.336 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:24.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:49:24.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:24.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:24.581 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:49:24.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:49:25.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:49:25.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:49:25.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:49:25.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:49:25.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:49:25.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:49:25.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:25.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:25.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:25.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:25.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:25.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:25.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:49:25.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:25.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:25.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-06-15T10:49:26.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:49:26.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:49:26.398 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:49:26.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:49:26.966 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:49:26.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:49:26.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:49:26.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:49:27.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-06-15T10:49:27.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:49:27.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:49:27.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:1+0 records in 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:1+0 records out 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000193835 s, 10.6 MB/s 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:49:27.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:49:27.209 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:49:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:49:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:49:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:49:28.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:49:28.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:49:28.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:49:30.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:49:30.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:49:30.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:49:30.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:49:30.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:49:30.291 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:49:30.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:49:30.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:49:30.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:49:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:49:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:49:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:49:30.324 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:30.326+0000 7faee0bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:30.327 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:30.330+0000 7faee0bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:30.328 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:30.330+0000 7faee0bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:30.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:30.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:30.905 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:30.906+0000 7faee0bf77c0 -1 Falling back to public interface 2024-06-15T10:49:31.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:31.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:31.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:49:31.841 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:31.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:31.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:32.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:33.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:33.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:33.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:33.426+0000 7faee0bf77c0 -1 osd.0 46 log_to_monitors true 2024-06-15T10:49:34.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:34.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:34.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:49:34.328 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:49:34.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:34.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 51 up_thru 51 down_at 47 last_clean_interval [5,46) [v2:127.0.0.1:6802/2543049576,v1:127.0.0.1:6803/2543049576] [v2:127.0.0.1:6804/2543049576,v1:127.0.0.1:6805/2543049576] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:49:34.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:34.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:34.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:34.710 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:34.710 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:34.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:34.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:34.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:34.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:34.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:34.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:34.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:34.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:34.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:35.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:35.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332098 2024-06-15T10:49:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332098 2024-06-15T10:49:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098' 2024-06-15T10:49:35.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:35.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626438 2024-06-15T10:49:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626438 2024-06-15T10:49:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098 1-180388626438' 2024-06-15T10:49:35.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:35.435 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-06-15T10:49:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-06-15T10:49:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332098 1-180388626438 2-64424509458' 2024-06-15T10:49:35.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:35.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-219043332098 2024-06-15T10:49:35.577 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:35.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:35.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-219043332098 2024-06-15T10:49:35.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332098 2024-06-15T10:49:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 219043332098' 2024-06-15T10:49:35.580 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 219043332098 2024-06-15T10:49:35.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332098 -lt 219043332098 2024-06-15T10:49:35.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:35.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626438 2024-06-15T10:49:35.868 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:35.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:49:35.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:35.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626438 2024-06-15T10:49:35.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626438 2024-06-15T10:49:35.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626438' 2024-06-15T10:49:35.872 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 180388626438 2024-06-15T10:49:35.872 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:36.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626438 -lt 180388626438 2024-06-15T10:49:36.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:36.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:36.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-06-15T10:49:36.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:49:36.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-06-15T10:49:36.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:36.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-06-15T10:49:36.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-06-15T10:49:36.128 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509458 2024-06-15T10:49:36.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:49:36.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-06-15T10:49:36.362 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:49:36.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:36.363 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:36.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:49:36.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:36.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:36.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:36.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:36.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:36.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:36.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:36.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:49:36.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:36.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:36.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:37.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:49:37.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:49:37.445 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:49:37.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:49:37.838 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:49:37.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:49:37.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:49:37.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:49:38.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:49:38.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:49:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:49:38.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:49:38.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:49:38.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:49:38.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:49:38.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:49:38.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:49:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-06-15T10:49:40.060 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:49:40.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:49:40.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:49:40.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:49:40.624 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:49:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:49:40.625 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:49:40.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:49:40.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:49:40.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:49:40.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:49:40.659 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:40.658+0000 7f3c0e3867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:40.661 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:40.662+0000 7f3c0e3867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:40.663 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:40.662+0000 7f3c0e3867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:49:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:49:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:49:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:49:40.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:49:40.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:40.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:49:40.865 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:49:40.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:40.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:41.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:41.230 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:41.230+0000 7f3c0e3867c0 -1 Falling back to public interface 2024-06-15T10:49:42.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:42.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:42.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:49:42.125 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:42.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:42.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:42.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:43.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:49:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:49:43.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:43.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:43.591 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:43.594+0000 7f3c0e3867c0 -1 osd.1 55 log_to_monitors true 2024-06-15T10:49:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:44.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:44.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:44.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:49:44.620 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:49:44.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:44.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:49:44.958 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 56 last_clean_interval [42,55) [v2:127.0.0.1:6810/4057154089,v1:127.0.0.1:6811/4057154089] [v2:127.0.0.1:6812/4057154089,v1:127.0.0.1:6813/4057154089] exists,up 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:44.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:44.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:45.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:45.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:45.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332101 2024-06-15T10:49:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332101 2024-06-15T10:49:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332101' 2024-06-15T10:49:45.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:45.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:45.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-06-15T10:49:45.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-06-15T10:49:45.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332101 1-257698037762' 2024-06-15T10:49:45.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:45.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:45.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-06-15T10:49:45.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-06-15T10:49:45.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-219043332101 1-257698037762 2-64424509461' 2024-06-15T10:49:45.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:45.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-219043332101 2024-06-15T10:49:45.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:45.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:45.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-219043332101 2024-06-15T10:49:45.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:45.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332101 2024-06-15T10:49:45.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 219043332101' 2024-06-15T10:49:45.809 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 219043332101 2024-06-15T10:49:45.809 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332101 -lt 219043332101 2024-06-15T10:49:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:46.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:46.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037762 2024-06-15T10:49:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:49:46.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037762 2024-06-15T10:49:46.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-06-15T10:49:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-06-15T10:49:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 257698037762 2024-06-15T10:49:46.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 257698037762 2024-06-15T10:49:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:47.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:49:47.297 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:47.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 257698037762 2024-06-15T10:49:47.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:49:48.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:49:48.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:49:48.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-06-15T10:49:48.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:48.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-06-15T10:49:48.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:48.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:49:48.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-06-15T10:49:48.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:48.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-06-15T10:49:48.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-15T10:49:48.790 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509461 2024-06-15T10:49:48.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:49:49.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-15T10:49:49.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:49:49.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:49.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:49.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:49:49.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:49:49.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:49:49.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:49:49.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:49:49.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:49:49.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:49:49.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:49:49.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-06-15T10:49:49.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:49:49.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:49:49.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:49:50.173 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:49:50.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:49:50.667 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:49:50.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:49:50.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:49:50.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:49:50.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-06-15T10:49:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:49:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:49:50.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-06-15T10:49:51.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-06-15T10:49:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-06-15T10:49:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:49:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:49:51.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:49:51.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:49:51.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:49:51.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:49:51.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:49:51.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:49:51.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:49:51.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:49:51.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:49:51.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:49:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:49:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:49:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:49:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:49:51.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:49:53.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:49:53.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:49:53.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:49:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:49:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:49:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:49:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:49:53.900 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:49:53.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:49:53.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:49:53.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:49:53.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:49:53.932 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:53.934+0000 7f91b15307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:53.934 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:53.934+0000 7f91b15307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:53.936 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:53.938+0000 7f91b15307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:49:54.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:49:54.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:54.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:54.485 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:54.486+0000 7f91b15307c0 -1 Falling back to public interface 2024-06-15T10:49:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:49:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:55.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:55.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:56.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:56.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:49:57.249 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:49:57.250+0000 7f91b15307c0 -1 osd.0 64 log_to_monitors true 2024-06-15T10:49:57.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:49:57.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:49:57.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:49:57.962 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:49:57.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:49:57.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:49:58.283 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 69 up_thru 69 down_at 65 last_clean_interval [51,64) [v2:127.0.0.1:6802/473020473,v1:127.0.0.1:6803/473020473] [v2:127.0.0.1:6804/473020473,v1:127.0.0.1:6805/473020473] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:49:58.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:49:58.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:49:58.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:49:58.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:49:58.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:49:58.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:49:58.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:49:58.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:49:58.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:49:58.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:49:58.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:49:58.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:49:58.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:49:58.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:49:58.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:49:58.820 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:49:58.820 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:49:58.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:49:58.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:58.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:49:59.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-06-15T10:49:59.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-06-15T10:49:59.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426' 2024-06-15T10:49:59.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:59.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:49:59.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-06-15T10:49:59.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-06-15T10:49:59.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426 1-257698037766' 2024-06-15T10:49:59.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:49:59.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:49:59.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-06-15T10:49:59.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-06-15T10:49:59.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743426 1-257698037766 2-64424509464' 2024-06-15T10:49:59.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:49:59.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743426 2024-06-15T10:49:59.376 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:49:59.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:49:59.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743426 2024-06-15T10:49:59.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:49:59.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-06-15T10:49:59.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743426' 2024-06-15T10:49:59.379 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 296352743426 2024-06-15T10:49:59.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:49:59.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 296352743426 2024-06-15T10:49:59.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:00.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:50:00.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:00.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743426 -lt 296352743426 2024-06-15T10:50:00.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:00.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037766 2024-06-15T10:50:00.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:00.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:00.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037766 2024-06-15T10:50:00.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:00.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-06-15T10:50:00.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037766' 2024-06-15T10:50:00.858 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 257698037766 2024-06-15T10:50:00.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037766 2024-06-15T10:50:01.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:01.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-06-15T10:50:01.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:01.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:01.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-06-15T10:50:01.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:01.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-06-15T10:50:01.111 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-06-15T10:50:01.111 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 64424509464 2024-06-15T10:50:01.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:01.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-06-15T10:50:01.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:01.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:01.367 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:01.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:01.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:01.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:01.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:01.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:01.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:01.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:50:02.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:50:02.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:50:02.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:50:02.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:50:02.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:50:02.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:50:02.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:50:02.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:50:02.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:50:02.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:50:04.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:50:04.672 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:50:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:50:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:50:04.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:50:04.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:50:04.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:50:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:50:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:50:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T10:50:04.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:50:04.677 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:50:04.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:50:04.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:50:04.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:50:04.678 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:50:04.709 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:04.710+0000 7fd15e25c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:04.711 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:04.714+0000 7fd15e25c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:04.713 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:04.714+0000 7fd15e25c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:04.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:50:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:50:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:05.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:05.258 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:05.258+0000 7fd15e25c7c0 -1 Falling back to public interface 2024-06-15T10:50:06.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:06.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:06.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:50:06.164 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:06.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:06.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:06.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:07.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:07.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:07.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:50:07.408 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:50:07.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:07.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:07.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:07.813 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:07.814+0000 7fd15e25c7c0 -1 osd.2 71 log_to_monitors true 2024-06-15T10:50:08.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:08.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:08.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:50:08.655 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:50:08.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:08.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 76 up_thru 76 down_at 72 last_clean_interval [15,71) [v2:127.0.0.1:6818/2172258962,v1:127.0.0.1:6819/2172258962] [v2:127.0.0.1:6820/2172258962,v1:127.0.0.1:6821/2172258962] exists,up 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:50:09.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:50:09.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:50:09.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:50:09.091 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:50:09.092 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:50:09.092 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:50:09.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:50:09.092 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:50:09.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:50:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:50:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:50:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:50:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:50:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:50:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:50:09.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:50:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:50:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:50:09.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:09.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:50:09.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743429 2024-06-15T10:50:09.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743429 2024-06-15T10:50:09.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429' 2024-06-15T10:50:09.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:09.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:50:09.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037769 2024-06-15T10:50:09.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037769 2024-06-15T10:50:09.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429 1-257698037769' 2024-06-15T10:50:09.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:09.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:50:09.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-06-15T10:50:09.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-06-15T10:50:09.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743429 1-257698037769 2-326417514498' 2024-06-15T10:50:09.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:09.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743429 2024-06-15T10:50:09.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:09.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:50:09.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743429 2024-06-15T10:50:09.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:09.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743429 2024-06-15T10:50:09.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743429' 2024-06-15T10:50:09.949 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 296352743429 2024-06-15T10:50:09.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743428 -lt 296352743429 2024-06-15T10:50:10.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:11.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:50:11.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:11.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743428 -lt 296352743429 2024-06-15T10:50:11.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:12.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:50:12.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:12.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743429 -lt 296352743429 2024-06-15T10:50:12.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:12.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037769 2024-06-15T10:50:12.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:12.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:12.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037769 2024-06-15T10:50:12.671 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:12.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037769 2024-06-15T10:50:12.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037769' 2024-06-15T10:50:12.673 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 257698037769 2024-06-15T10:50:12.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:12.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037769 -lt 257698037769 2024-06-15T10:50:12.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:12.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-06-15T10:50:12.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:12.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:12.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-06-15T10:50:12.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-06-15T10:50:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-06-15T10:50:12.914 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 326417514498 2024-06-15T10:50:12.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514498 -lt 326417514498 2024-06-15T10:50:13.162 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:13.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:13.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:13.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:13.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:13.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:13.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:13.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:13.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:13.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:13.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:13.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:13.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:13.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-06-15T10:50:13.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-06-15T10:50:13.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:50:13.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T10:50:13.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:50:13.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:50:13.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:50:13.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:50:13.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:50:14.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:50:14.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-06-15T10:50:16.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-06-15T10:50:16.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:50:16.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:50:16.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T10:50:16.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:50:16.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:50:16.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:50:16.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:50:16.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:50:16.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T10:50:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T10:50:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T10:50:16.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:50:16.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T10:50:16.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T10:50:16.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:50:16.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:50:16.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:50:16.278 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:16.278+0000 7f1751dbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:16.281 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:16.282+0000 7f1751dbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:16.282 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:16.282+0000 7f1751dbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:16.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:16.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:16.825 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:16.826+0000 7f1751dbb7c0 -1 Falling back to public interface 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:17.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:17.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:50:18.998 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:50:18.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:18.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:19.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:19.342 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:19.337+0000 7f1751dbb7c0 -1 osd.2 78 log_to_monitors true 2024-06-15T10:50:20.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:20.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:20.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:50:20.240 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:50:20.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:20.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6818/1128349757,v1:127.0.0.1:6819/1128349757] [v2:127.0.0.1:6820/1128349757,v1:127.0.0.1:6821/1128349757] exists,up 0ef308d4-ffb5-45b9-ac09-ecf7ef92e65e 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:50:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:50:20.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:50:20.612 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:50:20.612 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:50:20.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:50:20.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:50:20.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:50:20.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:21.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:50:21.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743432 2024-06-15T10:50:21.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743432 2024-06-15T10:50:21.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743432' 2024-06-15T10:50:21.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:21.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:50:21.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-06-15T10:50:21.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-06-15T10:50:21.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743432 1-257698037772' 2024-06-15T10:50:21.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:21.354 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:50:21.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-06-15T10:50:21.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-06-15T10:50:21.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743432 1-257698037772 2-356482285570' 2024-06-15T10:50:21.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:21.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743432 2024-06-15T10:50:21.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:21.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:50:21.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743432 2024-06-15T10:50:21.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743432 2024-06-15T10:50:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743432' 2024-06-15T10:50:21.495 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 296352743432 2024-06-15T10:50:21.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:21.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743431 -lt 296352743432 2024-06-15T10:50:21.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:22.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:50:22.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:22.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743432 -lt 296352743432 2024-06-15T10:50:22.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:22.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037772 2024-06-15T10:50:22.990 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:22.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:22.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037772 2024-06-15T10:50:22.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:22.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-06-15T10:50:22.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037772' 2024-06-15T10:50:22.995 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 257698037772 2024-06-15T10:50:22.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:23.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037772 -lt 257698037772 2024-06-15T10:50:23.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:23.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285570 2024-06-15T10:50:23.246 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:23.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:23.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285570 2024-06-15T10:50:23.249 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:23.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-06-15T10:50:23.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285570' 2024-06-15T10:50:23.250 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285570 2024-06-15T10:50:23.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:23.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-06-15T10:50:23.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:23.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:23.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:23.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:23.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:24.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:24.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:24.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:24.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:24.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:24.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:24.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:24.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-15T10:50:24.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:50:24.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-06-15T10:50:24.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-06-15T10:50:24.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:50:24.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:50:24.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-06-15T10:50:24.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:50:24.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:50:24.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:50:24.649 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:50:24.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:50:24.984 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:50:25.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:50:25.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:50:25.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:50:25.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-06-15T10:50:25.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:50:25.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:50:25.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-06-15T10:50:25.204 INFO:tasks.workunit.client.0.smithi116.stderr:2+0 records in 2024-06-15T10:50:25.204 INFO:tasks.workunit.client.0.smithi116.stderr:2+0 records out 2024-06-15T10:50:25.204 INFO:tasks.workunit.client.0.smithi116.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000138797 s, 29.5 MB/s 2024-06-15T10:50:25.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:50:25.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:50:25.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:50:25.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:50:25.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:50:25.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:50:25.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:50:25.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:50:25.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:50:25.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:50:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:50:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:50:27.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:50:27.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:50:27.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:50:27.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:50:27.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:50:27.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:50:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:50:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:50:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:50:27.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:27.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:50:27.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:50:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:50:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:50:27.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:50:27.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:50:27.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:50:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:50:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:50:27.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:50:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:27.753+0000 7fa4a0b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:27.756 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:27.757+0000 7fa4a0b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:27.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:27.757+0000 7fa4a0b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:28.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:50:28.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:50:28.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:50:28.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:50:28.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:50:28.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:28.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:50:28.008 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:50:28.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:28.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:28.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:28.317+0000 7fa4a0b707c0 -1 Falling back to public interface 2024-06-15T10:50:29.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:50:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:29.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:29.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:50:30.496 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:50:30.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:30.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:30.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:30.934 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:30.933+0000 7fa4a0b707c0 -1 osd.1 87 log_to_monitors true 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:31.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:32.099 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [60,87) [v2:127.0.0.1:6810/3842869531,v1:127.0.0.1:6811/3842869531] [v2:127.0.0.1:6812/3842869531,v1:127.0.0.1:6813/3842869531] exists,up 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:50:32.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:50:32.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:50:32.101 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:50:32.101 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:50:32.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:50:32.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:50:32.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:50:32.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:32.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:50:32.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743436 2024-06-15T10:50:32.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743436 2024-06-15T10:50:32.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436' 2024-06-15T10:50:32.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:32.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:50:32.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-06-15T10:50:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-06-15T10:50:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436 1-395136991234' 2024-06-15T10:50:32.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:32.866 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:50:33.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285573 2024-06-15T10:50:33.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285573 2024-06-15T10:50:33.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-296352743436 1-395136991234 2-356482285573' 2024-06-15T10:50:33.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:33.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-296352743436 2024-06-15T10:50:33.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:33.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:50:33.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-296352743436 2024-06-15T10:50:33.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743436 2024-06-15T10:50:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 296352743436' 2024-06-15T10:50:33.010 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 296352743436 2024-06-15T10:50:33.011 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:33.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743434 -lt 296352743436 2024-06-15T10:50:33.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:34.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:50:34.247 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:34.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743436 -lt 296352743436 2024-06-15T10:50:34.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:34.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-06-15T10:50:34.487 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:34.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:34.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:34.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-06-15T10:50:34.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-06-15T10:50:34.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-06-15T10:50:34.491 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 395136991234 2024-06-15T10:50:34.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:34.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-06-15T10:50:34.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:34.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:34.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285573 2024-06-15T10:50:34.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:34.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285573 2024-06-15T10:50:34.741 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:34.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285573 2024-06-15T10:50:34.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285573' 2024-06-15T10:50:34.742 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285573 2024-06-15T10:50:34.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:34.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285573 -lt 356482285573 2024-06-15T10:50:34.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:34.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:34.971 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:35.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:35.273 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:35.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:35.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:35.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:35.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:35.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:35.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:35.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:50:35.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:35.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:35.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:50:35.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:50:35.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:50:35.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:35.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:35.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:35.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:35.845 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:35.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:35.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:36.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:50:36.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:36.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:36.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:36.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:50:36.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:50:36.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:50:36.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:50:36.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:50:36.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:50:36.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:50:36.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:50:36.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:50:36.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:50:36.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:50:36.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:50:36.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:50:36.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:50:36.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:50:36.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:36.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:36.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:36.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:36.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:36.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:36.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:37.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T10:50:37.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:37.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:37.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:37.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-06-15T10:50:37.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:50:37.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:50:37.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:50:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:50:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:50:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:50:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:50:37.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:50:37.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:50:37.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:37.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:37.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:37.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:37.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:37.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:37.937 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:38.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:38.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:38.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:38.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-06-15T10:50:38.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:50:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:50:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-06-15T10:50:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:50:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:50:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:50:38.779 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:50:38.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:50:39.139 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:50:39.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:50:39.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:50:39.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:50:39.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-06-15T10:50:39.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:50:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:50:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:50:39.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:50:39.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-06-15T10:50:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:50:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:50:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:50:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:50:41.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:50:41.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:50:41.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:50:41.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:50:41.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:50:41.770 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:50:41.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:50:41.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:50:41.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:50:41.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:50:41.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:50:41.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:50:41.804 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:41.805+0000 7fd740f527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:41.810 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:41.809+0000 7fd740f527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:41.811 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:41.813+0000 7fd740f527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:42.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:50:42.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:42.350 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:42.349+0000 7fd740f527c0 -1 Falling back to public interface 2024-06-15T10:50:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:50:43.260 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:43.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:43.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:50:43.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:50:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:50:44.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:44.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:50:44.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:45.067 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:45.069+0000 7fd740f527c0 -1 osd.0 96 log_to_monitors true 2024-06-15T10:50:45.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:45.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:45.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:50:45.750 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:50:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:50:46.293 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [69,96) [v2:127.0.0.1:6802/2784688324,v1:127.0.0.1:6803/2784688324] [v2:127.0.0.1:6804/2784688324,v1:127.0.0.1:6805/2784688324] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:50:46.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:50:46.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:50:46.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:50:46.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:50:46.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:50:46.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:50:46.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:50:46.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:50:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:50:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:46.764 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:50:46.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:50:46.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:46.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:50:46.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-06-15T10:50:46.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-06-15T10:50:46.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898' 2024-06-15T10:50:46.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:46.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:50:47.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991238 2024-06-15T10:50:47.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991238 2024-06-15T10:50:47.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238' 2024-06-15T10:50:47.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:47.125 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:50:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285577 2024-06-15T10:50:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285577 2024-06-15T10:50:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238 2-356482285577' 2024-06-15T10:50:47.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:47.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696898 2024-06-15T10:50:47.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:47.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:50:47.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696898 2024-06-15T10:50:47.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:47.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-06-15T10:50:47.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-06-15T10:50:47.269 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 433791696898 2024-06-15T10:50:47.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:47.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 433791696898 2024-06-15T10:50:47.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:50:48.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:50:48.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-06-15T10:50:48.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:48.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991238 2024-06-15T10:50:48.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:48.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:48.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991238 2024-06-15T10:50:48.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:48.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991238 2024-06-15T10:50:48.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991238' 2024-06-15T10:50:48.759 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 395136991238 2024-06-15T10:50:48.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:49.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991238 -lt 395136991238 2024-06-15T10:50:49.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:49.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285577 2024-06-15T10:50:49.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:49.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:49.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285577 2024-06-15T10:50:49.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:49.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285577 2024-06-15T10:50:49.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285577' 2024-06-15T10:50:49.034 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285577 2024-06-15T10:50:49.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:49.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285577 -lt 356482285577 2024-06-15T10:50:49.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:49.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:49.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:49.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:49.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:49.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:49.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:49.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:49.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:49.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:49.549 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:49.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:49.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:49.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:49.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:50:50.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:50:50.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:50:50.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:50:50.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:50:50.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:50:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:50:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:50:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:50:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:50:50.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-06-15T10:50:52.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:50:52.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:50:52.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:50:52.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:50:52.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:50:52.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:50:52.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:50:52.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:50:52.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:50:52.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:50:52.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:50:52.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:50:52.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:50:52.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:50:52.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:50:52.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:50:52.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:50:52.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:50:52.385 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:50:52.385 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:50:52.386 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:50:52.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:52.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:50:52.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:50:52.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:50:52.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:50:52.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:50:52.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:50:52.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:50:52.388 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:50:52.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:50:52.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:50:52.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:50:52.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:50:52.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:50:52.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:50:52.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:52.417+0000 7f0dbdccf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:52.418 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:52.417+0000 7f0dbdccf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:52.419 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:52.421+0000 7f0dbdccf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:52.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:50:52.637 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:50:52.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:52.637 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:52.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:52.953 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:52.953+0000 7f0dbdccf7c0 -1 Falling back to public interface 2024-06-15T10:50:53.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:53.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:53.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:50:53.880 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:53.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:53.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:54.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:55.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:55.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:55.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:50:55.122 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:50:55.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:55.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:55.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:50:55.447 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:50:55.449+0000 7f0dbdccf7c0 -1 osd.1 103 log_to_monitors true 2024-06-15T10:50:56.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:50:56.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:50:56.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:50:56.355 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:50:56.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:50:56.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/3204058852,v1:127.0.0.1:6811/3204058852] [v2:127.0.0.1:6812/3204058852,v1:127.0.0.1:6813/3204058852] exists,up 1c943628-6725-40ce-8d8a-c9d5aec6c772 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:50:56.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:50:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:50:56.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:50:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:50:56.664 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:50:56.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:50:56.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:50:56.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:50:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:50:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:50:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:50:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:50:56.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:50:56.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:50:57.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:50:57.118 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:50:57.118 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:50:57.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:50:57.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:57.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:50:57.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-06-15T10:50:57.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-06-15T10:50:57.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901' 2024-06-15T10:50:57.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:57.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:50:57.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-06-15T10:50:57.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-06-15T10:50:57.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-06-15T10:50:57.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:50:57.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:50:57.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285580 2024-06-15T10:50:57.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285580 2024-06-15T10:50:57.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-356482285580' 2024-06-15T10:50:57.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:57.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696901 2024-06-15T10:50:57.573 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:57.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:50:57.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696901 2024-06-15T10:50:57.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:57.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-06-15T10:50:57.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-06-15T10:50:57.575 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 433791696901 2024-06-15T10:50:57.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:50:57.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-06-15T10:50:57.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:57.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-06-15T10:50:57.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:57.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:50:57.827 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-06-15T10:50:57.828 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:57.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-06-15T10:50:57.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-06-15T10:50:57.830 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 463856467970 2024-06-15T10:50:57.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:50:58.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-06-15T10:50:58.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:50:58.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285580 2024-06-15T10:50:58.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:50:58.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:50:58.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285580 2024-06-15T10:50:58.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:50:58.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285580 2024-06-15T10:50:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285580' 2024-06-15T10:50:58.086 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285580 2024-06-15T10:50:58.086 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:50:58.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285580 -lt 356482285580 2024-06-15T10:50:58.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:50:58.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:58.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:50:58.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:50:58.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:50:58.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:50:58.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:50:58.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:50:58.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:50:58.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:50:59.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:50:59.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:50:59.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:50:59.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:50:59.493 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:50:59.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:50:59.895 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:50:59.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:50:59.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:50:59.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:51:00.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-06-15T10:51:00.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-06-15T10:51:00.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-06-15T10:51:00.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-15T10:51:00.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-06-15T10:51:00.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-06-15T10:51:00.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:51:00.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:51:00.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:51:00.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:51:00.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:51:00.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-06-15T10:51:02.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:51:02.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:51:02.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:51:02.664 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:51:02.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:51:02.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:51:02.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:51:02.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:51:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:51:02.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:51:02.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:51:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:51:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:51:02.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:51:02.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:51:02.707 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:02.709+0000 7f5b240aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:02.710 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:02.709+0000 7f5b240aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:02.712 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:02.713+0000 7f5b240aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:51:02.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:02.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:03.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:03.257 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:03.257+0000 7f5b240aa7c0 -1 Falling back to public interface 2024-06-15T10:51:04.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:04.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:04.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:51:04.154 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:51:04.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:04.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:04.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:51:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:51:05.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:05.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:05.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:05.748 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:05.749+0000 7f5b240aa7c0 -1 osd.0 112 log_to_monitors true 2024-06-15T10:51:06.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:06.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:06.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:51:06.628 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:51:06.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:06.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:06.964 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 117 up_thru 117 down_at 113 last_clean_interval [101,112) [v2:127.0.0.1:6802/930846873,v1:127.0.0.1:6803/930846873] [v2:127.0.0.1:6804/930846873,v1:127.0.0.1:6805/930846873] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:51:06.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:51:06.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:51:06.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:51:06.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:51:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:51:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:51:06.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:51:06.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:51:06.966 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:51:06.966 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:51:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:51:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:51:06.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:51:07.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:51:07.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:51:07.382 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:51:07.382 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:51:07.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:51:07.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:07.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:51:07.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-06-15T10:51:07.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-06-15T10:51:07.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634' 2024-06-15T10:51:07.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:07.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:51:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467973 2024-06-15T10:51:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467973 2024-06-15T10:51:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973' 2024-06-15T10:51:07.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:07.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:51:07.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285583 2024-06-15T10:51:07.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285583 2024-06-15T10:51:07.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973 2-356482285583' 2024-06-15T10:51:07.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:07.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-502511173634 2024-06-15T10:51:07.810 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:07.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:51:07.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-502511173634 2024-06-15T10:51:07.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:07.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-06-15T10:51:07.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 502511173634' 2024-06-15T10:51:07.814 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 502511173634 2024-06-15T10:51:07.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:51:08.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-06-15T10:51:08.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:08.034 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467973 2024-06-15T10:51:08.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:08.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:51:08.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467973 2024-06-15T10:51:08.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:08.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467973 2024-06-15T10:51:08.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467973' 2024-06-15T10:51:08.039 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 463856467973 2024-06-15T10:51:08.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:51:08.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467973 -lt 463856467973 2024-06-15T10:51:08.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:08.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285583 2024-06-15T10:51:08.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:08.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:51:08.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285583 2024-06-15T10:51:08.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:08.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285583 2024-06-15T10:51:08.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285583' 2024-06-15T10:51:08.295 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285583 2024-06-15T10:51:08.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:51:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285582 -lt 356482285583 2024-06-15T10:51:08.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:51:09.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:51:09.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:51:09.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285582 -lt 356482285583 2024-06-15T10:51:09.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:51:10.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:51:10.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:51:11.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285584 -lt 356482285583 2024-06-15T10:51:11.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:51:11.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:51:11.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:51:11.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:51:11.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:51:11.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:51:11.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:51:11.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:51:11.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:51:11.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:51:11.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:51:11.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:51:11.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:51:11.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:51:11.524 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:51:11.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:51:11.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:51:12.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:51:12.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:51:14.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:51:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:51:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:51:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:51:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:51:14.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:14.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:51:14.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:51:14.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:51:14.373 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:51:14.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:51:14.374 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:51:14.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:51:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:51:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:51:14.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:51:14.407 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:14.405+0000 7fd920db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:14.410 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:14.409+0000 7fd920db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:14.412 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:14.413+0000 7fd920db27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:14.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:14.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:14.977 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:14.977+0000 7fd920db27c0 -1 Falling back to public interface 2024-06-15T10:51:15.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:15.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:51:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:51:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:15.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:16.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:17.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:17.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:17.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:51:17.159 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:51:17.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:17.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:17.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:17.437 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:17.437+0000 7fd920db27c0 -1 osd.0 119 log_to_monitors true 2024-06-15T10:51:18.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:18.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:18.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:51:18.403 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:51:18.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:18.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 124 up_thru 124 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6802/1087808054,v1:127.0.0.1:6803/1087808054] [v2:127.0.0.1:6804/1087808054,v1:127.0.0.1:6805/1087808054] exists,up 6b38592c-8df4-4003-ba9e-814328cbd2f2 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:51:18.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:51:18.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:51:18.747 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:51:18.747 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:51:18.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:51:18.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:51:18.747 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:51:18.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:51:18.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:51:18.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:51:18.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:51:18.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:51:18.870 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:51:19.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:51:19.216 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:51:19.216 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T10:51:19.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:51:19.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:19.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:51:19.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=532575944706 2024-06-15T10:51:19.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 532575944706 2024-06-15T10:51:19.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706' 2024-06-15T10:51:19.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:19.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:51:19.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467976 2024-06-15T10:51:19.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467976 2024-06-15T10:51:19.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467976' 2024-06-15T10:51:19.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:51:19.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T10:51:19.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285587 2024-06-15T10:51:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285587 2024-06-15T10:51:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467976 2-356482285587' 2024-06-15T10:51:19.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:19.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-532575944706 2024-06-15T10:51:19.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:19.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:51:19.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-532575944706 2024-06-15T10:51:19.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=532575944706 2024-06-15T10:51:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 532575944706' 2024-06-15T10:51:19.647 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 532575944706 2024-06-15T10:51:19.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:51:19.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 532575944706 -lt 532575944706 2024-06-15T10:51:19.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:19.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467976 2024-06-15T10:51:19.879 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:19.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:51:19.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467976 2024-06-15T10:51:19.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:19.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467976 2024-06-15T10:51:19.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467976' 2024-06-15T10:51:19.883 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 463856467976 2024-06-15T10:51:19.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:51:20.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467976 -lt 463856467976 2024-06-15T10:51:20.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:51:20.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285587 2024-06-15T10:51:20.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:51:20.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T10:51:20.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285587 2024-06-15T10:51:20.135 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:51:20.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285587 2024-06-15T10:51:20.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285587' 2024-06-15T10:51:20.137 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.2 seq 356482285587 2024-06-15T10:51:20.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T10:51:20.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285587 -lt 356482285587 2024-06-15T10:51:20.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:51:20.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:51:20.390 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:51:20.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:51:20.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:51:20.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:51:20.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:51:20.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:51:20.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:51:20.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:51:20.702 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:51:20.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:51:20.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:51:20.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:51:20.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:51:21.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:51:21.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:51:21.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:51:21.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-06-15T10:51:21.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-06-15T10:51:21.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T10:51:21.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-06-15T10:51:21.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-06-15T10:51:21.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:51:21.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-06-15T10:51:21.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-06-15T10:51:21.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-15T10:51:21.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-15T10:51:21.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:21.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:21.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:21.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:21.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:21.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-15T10:51:21.917 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:51:21.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:21.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:21.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:21.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:21.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:21.935 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:22.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:22.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:23.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:23.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:23.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:23.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:23.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:23.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:23.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:23.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:23.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:24.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:24.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:24.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:24.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:24.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:24.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:24.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:24.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:24.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:25.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:25.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:25.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:25.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:25.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:25.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:25.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:25.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:25.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:26.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:26.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:26.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:26.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:26.892 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:26.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:26.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:27.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:27.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:28.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:28.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:28.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:51:28.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:28.136 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:51:28.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:28.137 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:51:28.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:51:22.381596+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:28.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:51:28.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-15T10:51:28.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:51:28.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-06-15T10:51:28.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:51:28.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-15T10:51:28.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-15T10:51:28.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:51:28.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=124 2024-06-15T10:51:28.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:51:28.514 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:51:28.514 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:51:28.514 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:51:28.514 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:51:28.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:51:28.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:51:28.555 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:51:28.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:51:28.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:51:28.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:28.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:51:28.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:28.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:51:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:28.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-15T10:51:29.035 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-15T10:49:01.511137+0000 last_deep_scrub_stamp 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:29.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:51:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:29.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:51:29.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:29.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:30.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:30.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:30.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:51:30.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:30.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:51:30.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:30.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:51:30.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:49:01.511137+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:30.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:51:31.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:51:31.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:51:31.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:51:31.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:51:31.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:51:31.513 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:51:31.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:51:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:51:29.624735+0000 '>' 2024-06-15T10:49:01.511137+0000 2024-06-15T10:51:31.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:51:31.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-06-15T10:51:31.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:51:31.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-06-15T10:51:31.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:51:31.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-06-15T10:51:31.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-06-15T10:51:31.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:51:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=124 2024-06-15T10:51:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-06-15T10:51:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:51:31.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:51:31.949 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:51:31.950 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:51:31.950 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:51:31.950 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:51:31.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-06-15T10:51:31.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:51:31.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:51:31.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:51:31.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-06-15T10:51:31.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-06-15T10:51:31.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-06-15T10:51:32.320 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' removed 2024-06-15T10:51:32.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:51:32.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:51:32.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:51:32.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:51:32.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:51:32.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:51:32.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:51:32.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:51:32.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:51:32.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:51:32.655 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:51:32.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:51:32.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:51:32.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:51:32.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:51:32.659 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:51:32.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:51:32.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:51:32.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:51:32.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:51:32.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:51:32.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:51:32.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:51:32.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:51:32.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:32.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:32.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:51:32.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:51:32.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:51:32.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:51:32.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:51:32.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:51:32.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:51:32.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:51:32.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:51:32.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:51:32.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:51:32.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:51:32.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:51:32.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:51:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:51:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:51:32.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:51:32.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:51:32.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:51:32.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:51:32.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:51:32.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:51:32.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:32.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:32.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:51:32.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:51:32.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:51:32.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:51:32.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:51:32.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:32.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:32.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:51:32.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:51:32.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:51:32.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:51:32.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:51:32.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-15T10:51:33.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:51:33.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:51:33.097 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:51:33.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:51:33.097 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:33.098 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:33.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-15T10:51:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:51:33.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:51:33.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:51:33.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:51:33.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:51:33.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:51:33.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:51:33.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:51:33.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:51:33.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:51:33.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:51:33.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:51:33.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:51:33.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:51:33.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:51:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:51:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:51:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:51:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:51:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:33.438 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:51:33.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:51:33.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:51:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-15T10:51:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:51:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:51:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:51:33.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:51:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:51:33.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:51:34.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:51:34.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:51:34.407 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:51:34.407 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:51:34.407 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:34.408 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:34.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:34.408 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:51:34.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:51:34.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:51:34.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:51:34.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:51:34.451 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:51:34.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:51:34.452 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:51:34.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9b098f5b-3fdb-47fa-97cc-efb99630c9df' 2024-06-15T10:51:34.452 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:51:34.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2cm1m/qZKHBAAdONdtVt1aRDxNpmZQR/fiA== 2024-06-15T10:51:34.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2cm1m/qZKHBAAdONdtVt1aRDxNpmZQR/fiA=="}' 2024-06-15T10:51:34.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b098f5b-3fdb-47fa-97cc-efb99630c9df -i td/osd-scrub-repair/0/new.json 2024-06-15T10:51:34.710 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:51:34.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:51:34.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2cm1m/qZKHBAAdONdtVt1aRDxNpmZQR/fiA== --osd-uuid 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:51:34.762 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.760+0000 7f6a1b4d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:34.764 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.764+0000 7f6a1b4d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.764+0000 7f6a1b4d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:34.765 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.764+0000 7f6a1b4d57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:51:34.766 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.764+0000 7f6a1b4d57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:51:34.766 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.764+0000 7f6a1b4d57c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:51:34.852 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:34.852+0000 7f6a1b4d57c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:51:40.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:51:40.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:51:40.562 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:51:40.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:51:40.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:51:40.850 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:51:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:51:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:51:40.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:51:40.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:51:40.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:51:40.881 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:40.880+0000 7f81d88977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:40.883 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:40.884+0000 7f81d88977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:40.884 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:40.884+0000 7f81d88977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:51:41.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:51:41.115 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:51:41.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:41.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:41.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:41.442 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:41.440+0000 7f81d88977c0 -1 Falling back to public interface 2024-06-15T10:51:42.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:42.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:42.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:51:42.365 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:51:42.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:42.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:42.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:43.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:43.607 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:51:43.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:43.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:51:43.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:43.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:43.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:43.874 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:51:43.872+0000 7f81d88977c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:51:44.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:44.846 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:44.847 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:51:44.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:51:44.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:44.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:45.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:46.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:46.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:46.088 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:51:46.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:51:46.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:46.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:46.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:47.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:47.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:47.332 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:51:47.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:51:47.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:47.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:48.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:48.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:48.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:51:48.653 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:51:48.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:48.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:48.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:49.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:49.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:49.894 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:51:49.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:51:49.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:49.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:50.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:51.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:51.136 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:51:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:51:51.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:51.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:51.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:52.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:52.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:52.380 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:51:52.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:51:52.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:52.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:52.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:53.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:53.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:53.692 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:51:53.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:51:53.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:53.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:53.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:54.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:54.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:54.936 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:51:54.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:51:54.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:54.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:55.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:56.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:56.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:56.181 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:51:56.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:51:56.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:56.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:56.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:57.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:57.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:57.638 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:51:57.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:51:57.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:57.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:57.954 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:51:58.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:51:58.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:51:58.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:51:58.956 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:51:58.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:51:58.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:51:59.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:00.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:00.626 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:52:00.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:52:00.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:00.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:52:01.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:02.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:02.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:02.012 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:52:02.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:52:02.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:02.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:52:02.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:03.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:03.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:03.715 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:52:03.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:52:03.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:03.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:52:04.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:05.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:05.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:05.369 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:52:05.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:52:05.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:05.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:52:05.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:06.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:06.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:06.654 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:52:06.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:52:06.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:06.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:52:06.921 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2208049024,v1:127.0.0.1:6803/2208049024] [v2:127.0.0.1:6804/2208049024,v1:127.0.0.1:6805/2208049024] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:52:06.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:52:06.926 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:52:06.927 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:52:06.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:52:06.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:52:06.928 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:52:06.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:52:06.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e86e26b4-2c61-4754-b772-63f282ebe161' 2024-06-15T10:52:06.929 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:52:06.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:52:06.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWcm1mpk/bOBAAgqITUGEpEPPmk2waVq5jDw== 2024-06-15T10:52:06.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWcm1mpk/bOBAAgqITUGEpEPPmk2waVq5jDw=="}' 2024-06-15T10:52:06.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e86e26b4-2c61-4754-b772-63f282ebe161 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:52:07.225 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:52:07.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:52:07.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWcm1mpk/bOBAAgqITUGEpEPPmk2waVq5jDw== --osd-uuid e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:52:07.281 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.280+0000 7f6b83f3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:07.305 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.304+0000 7f6b83f3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:07.306 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.304+0000 7f6b83f3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:07.306 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.304+0000 7f6b83f3f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:52:07.306 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.304+0000 7f6b83f3f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:52:07.307 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.304+0000 7f6b83f3f7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:52:07.372 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:07.372+0000 7f6b83f3f7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:52:13.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:52:13.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:52:13.403 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:52:13.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:52:13.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:52:13.681 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:52:13.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:52:13.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:52:13.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:52:13.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:52:13.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:52:13.713 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:13.712+0000 7fa3a7a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:13.715 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:13.716+0000 7fa3a7a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:13.716 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:13.716+0000 7fa3a7a887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:52:13.930 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:52:13.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:13.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:14.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:14.258 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:14.256+0000 7fa3a7a887c0 -1 Falling back to public interface 2024-06-15T10:52:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:15.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:15.180 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:52:15.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:52:15.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:15.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:15.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:16.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:16.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:16.427 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:52:16.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:52:16.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:16.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:16.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:16.827 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:52:16.823+0000 7fa3a7a887c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:52:17.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:17.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:17.681 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:52:17.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:52:17.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:17.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:17.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:18.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:18.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:18.988 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:52:18.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:52:18.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:18.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:19.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:20.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:20.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:20.231 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:52:20.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:52:20.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:20.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:20.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:21.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:21.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:21.507 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:52:21.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:52:21.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:21.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:21.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:22.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:22.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:22.806 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:52:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:52:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:22.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:23.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:24.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:24.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:24.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:52:24.050 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:52:24.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:24.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:24.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:25.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:25.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:25.289 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:52:25.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:52:25.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:25.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:25.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:26.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:26.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:26.670 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:52:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:52:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:26.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:27.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:27.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:27.912 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:52:27.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:52:27.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:27.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:28.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:29.153 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:52:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:52:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:29.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:52:30.396 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:52:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:30.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:31.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:32.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:32.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:32.159 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:52:32.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:52:32.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:32.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:32.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:33.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:33.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:33.615 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:52:33.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:52:33.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:33.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:34.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:35.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:35.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:35.401 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:52:35.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:52:35.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:35.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:35.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:36.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:36.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:36.966 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:52:36.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:52:36.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:36.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:38.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:39.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:39.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:39.006 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:52:39.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:52:39.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:39.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:39.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:52:40.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:52:40.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:52:40.462 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:52:40.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:52:40.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:52:40.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:52:40.740 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1164991086,v1:127.0.0.1:6811/1164991086] [v2:127.0.0.1:6812/1164991086,v1:127.0.0.1:6813/1164991086] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:52:40.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:52:40.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:52:40.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:52:40.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-06-15T10:52:40.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:52:40.965 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:52:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:52:40.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:52:41.340 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:52:41.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:52:42.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:52:44.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-15T10:52:44.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:52:44.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:52:44.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:52:44.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:52:44.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:52:44.021 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:52:44.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:52:44.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:52:44.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:52:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:52:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:52:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:52:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:52:44.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:52:44.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:52:44.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:52:44.392 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:52:44.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:52:44.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:52:44.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:52:44.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-06-15T10:52:44.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-06-15T10:52:44.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-06-15T10:52:44.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:52:44.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:52:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-06-15T10:52:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-06-15T10:52:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672964' 2024-06-15T10:52:44.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:52:44.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-06-15T10:52:44.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:52:44.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:52:44.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-06-15T10:52:44.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:52:44.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-06-15T10:52:44.725 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836495 2024-06-15T10:52:44.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-06-15T10:52:44.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:52:44.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-06-15T10:52:44.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:52:44.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-06-15T10:52:44.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:52:44.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:52:44.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-06-15T10:52:44.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:52:44.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-06-15T10:52:44.953 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672964 2024-06-15T10:52:44.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-06-15T10:52:44.953 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:52:45.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672964 2024-06-15T10:52:45.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:52:46.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:52:46.234 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:52:46.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672964 2024-06-15T10:52:46.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:52:47.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:52:47.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:52:47.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-06-15T10:52:47.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:52:47.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:52:47.723 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:52:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:52:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:52:48.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:52:48.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:52:48.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:52:48.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:52:48.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:52:48.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:52:48.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:52:48.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:52:48.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-06-15T10:52:48.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-15T10:52:48.961 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' created 2024-06-15T10:52:48.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:52:49.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-06-15T10:52:49.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-15T10:52:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'csr_pool' created 2024-06-15T10:52:50.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:52:51.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-15T10:52:51.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:52:51.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:52:51.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:52:51.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:52:51.338 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:52:51.339 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:52:51.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:52:51.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:52:51.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:52:51.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:52:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:52:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:52:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:52:51.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:52:51.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:52:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:52:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:52:51.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:52:51.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:52:51.833 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:52:51.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-06-15T10:52:51.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-06-15T10:52:51.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-06-15T10:52:51.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:52:51.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:52:52.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-15T10:52:52.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-15T10:52:52.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672967' 2024-06-15T10:52:52.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:52:52.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:52:52.118 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-06-15T10:52:52.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:52:52.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-06-15T10:52:52.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:52:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-06-15T10:52:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-06-15T10:52:52.122 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836498 2024-06-15T10:52:52.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:52:52.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836498 2024-06-15T10:52:52.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:52:53.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:52:53.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:52:53.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-06-15T10:52:53.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:52:53.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:52:53.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-15T10:52:53.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:52:53.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-15T10:52:53.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:52:53.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-15T10:52:53.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-15T10:52:53.622 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672967 2024-06-15T10:52:53.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:52:53.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-15T10:52:53.884 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:52:53.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:52:53.885 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:52:54.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:52:54.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:52:54.156 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:52:54.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:52:54.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:52:54.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:52:54.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:52:54.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:52:54.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:52:54.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:52:54.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-15T10:52:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:52:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-15T10:52:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-15T10:52:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:52:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:52:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-15T10:52:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:52:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:52:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:52:55.045 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:52:57.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:52:57.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:52:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:52:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-15T10:52:57.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:52:57.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:52:57.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:52:57.655 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:52:57.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:52:58.014 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:52:58.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:52:58.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:52:58.033 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:52:58.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-06-15T10:52:58.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-06-15T10:52:58.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:52:58.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:52:58.885 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:52:58.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:52:59.235 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:52:59.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:52:59.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:52:59.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:52:59.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-06-15T10:52:59.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:52:59.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:52:59.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:00.114 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:00.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:00.633 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:00.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:00.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:00.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:00.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-06-15T10:53:01.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-06-15T10:53:01.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:01.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-15T10:53:01.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:01.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:01.502 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:01.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:01.879 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:01.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:01.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:01.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:02.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-06-15T10:53:02.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-06-15T10:53:02.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:02.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:02.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:02.757 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:02.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:03.226 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:03.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:03.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:03.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:03.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-06-15T10:53:03.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:03.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:03.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:04.055 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:04.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:04.574 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:04.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:04.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:04.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:04.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-06-15T10:53:04.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-06-15T10:53:05.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:05.113 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:05.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:05.451 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:05.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:05.853 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:05.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:05.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:05.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:06.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-06-15T10:53:06.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-06-15T10:53:06.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:06.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:06.960 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:06.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:07.324 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:07.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:07.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:07.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-06-15T10:53:07.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:07.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:08.177 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:08.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:08.713 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:08.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-06-15T10:53:09.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-06-15T10:53:09.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:09.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:09.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:09.649 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:09.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:10.143 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:10.167 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:10.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-06-15T10:53:10.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-06-15T10:53:10.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:10.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:10.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:10.996 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:11.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:11.406 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:11.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:11.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:11.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:11.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-06-15T10:53:11.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:11.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:12.309 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:12.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:12.839 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:12.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:13.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-06-15T10:53:13.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-06-15T10:53:13.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:13.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:13.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:13.767 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:13.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:14.160 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:14.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:14.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:14.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:14.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-06-15T10:53:14.507 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-06-15T10:53:14.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:14.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-15T10:53:14.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:14.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:15.005 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:15.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:15.390 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:15.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:15.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:15.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:15.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-06-15T10:53:15.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-06-15T10:53:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:15.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:16.268 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:16.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:16.661 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:16.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:16.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:16.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:16.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-06-15T10:53:17.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:17.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:17.548 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:17.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:18.058 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:18.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:18.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-06-15T10:53:18.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-06-15T10:53:18.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:18.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-15T10:53:18.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:53:18.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:53:18.987 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:53:19.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:53:19.364 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:53:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:53:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:53:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:53:19.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-06-15T10:53:19.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-06-15T10:53:19.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-06-15T10:53:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:1025+0 records in 2024-06-15T10:53:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:1025+0 records out 2024-06-15T10:53:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.0034835 s, 301 MB/s 2024-06-15T10:53:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-06-15T10:53:20.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-06-15T10:53:20.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-15T10:53:20.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-15T10:53:20.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-15T10:53:20.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-15T10:53:20.142 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:53:20.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-15T10:53:20.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-06-15T10:53:20.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-15T10:53:20.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-15T10:53:20.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-15T10:53:20.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:53:20.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:20.644 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:20.645 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:53:20.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-15T10:53:20.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.62093/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-15T10:53:20.756 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T10:53:20.756 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-15T10:53:20.756 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T10:53:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-06-15T10:53:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:53:20.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:53:20.770 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:53:20.771 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:20.771 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:20.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:53:20.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-06-15T10:53:20.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-06-15T10:53:20.880 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T10:53:20.881 INFO:tasks.workunit.client.0.smithi116.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-06-15T10:53:20.881 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T10:53:20.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-15T10:53:20.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-15T10:53:20.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:20.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:20.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:20.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:20.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:21.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:21.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-15T10:53:21.348 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-15T10:52:50.266710+0000 last_deep_scrub_stamp 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:53:21.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:21.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:21.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:21.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:21.365 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:21.366 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:21.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:52:50.266710+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:21.599 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:53:22.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:53:22.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:22.601 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:22.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:22.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:22.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:22.602 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:22.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:52:50.266710+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:22.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:53:23.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:53:23.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:23.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:23.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:23.843 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:23.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:23.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:24.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:52:50.266710+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:24.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:53:25.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:53:25.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:25.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:25.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:25.084 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:25.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:25.085 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:25.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:52:50.266710+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:25.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:53:26.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:53:26.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:26.330 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:26.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:26.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:26.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:26.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:26.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:52:50.266710+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:26.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:53:27.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:53:27.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:53:27.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:53:27.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:53:27.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:53:27.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:53:27.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:53:27.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:52:50.266710+0000 2024-06-15T10:53:27.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:53:27.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-06-15T10:53:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:27.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-06-15T10:53:27.812 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-06-15T10:53:27.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:53:27.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:53:27.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:27.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:53:27.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:53:28.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:53:28.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:29.389 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-06-15T10:53:29.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:53:29.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:53:29.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:53:29.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:53:29.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:53:29.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:53:29.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:53:29.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:53:29.968 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:53:29.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:53:29.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:53:29.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:53:29.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:53:29.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:53:29.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:53:30.001 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:30.002+0000 7fcbc1a767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:30.004 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:30.006+0000 7fcbc1a767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:30.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:30.006+0000 7fcbc1a767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:30.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:30.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:30.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:30.558 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:30.558+0000 7fcbc1a767c0 -1 Falling back to public interface 2024-06-15T10:53:31.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:31.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:31.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:53:31.467 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:53:31.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:31.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:31.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:32.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:53:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:53:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:32.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:33.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:33.302 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:33.302+0000 7fcbc1a767c0 -1 osd.1 64 log_to_monitors true 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:34.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:34.377 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/1662983331,v1:127.0.0.1:6811/1662983331] [v2:127.0.0.1:6812/1662983331,v1:127.0.0.1:6813/1662983331] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:53:34.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:53:34.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:53:34.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:53:34.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:53:34.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:53:34.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:53:34.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:53:34.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:53:34.379 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:53:34.379 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:53:34.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:53:34.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:53:34.379 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:53:34.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:53:34.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:53:34.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:53:34.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:53:34.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:53:34.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:53:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:53:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:53:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:53:34.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:34.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:53:34.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-06-15T10:53:34.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-06-15T10:53:34.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-06-15T10:53:34.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:34.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:53:35.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-06-15T10:53:35.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-06-15T10:53:35.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-292057776130' 2024-06-15T10:53:35.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:35.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-06-15T10:53:35.093 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:35.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:53:35.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-06-15T10:53:35.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:35.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-06-15T10:53:35.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-06-15T10:53:35.096 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836507 2024-06-15T10:53:35.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:35.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-15T10:53:35.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:53:36.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:53:36.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:36.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-06-15T10:53:36.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:53:37.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:53:37.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:37.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836507 2024-06-15T10:53:37.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:37.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776130 2024-06-15T10:53:37.794 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:37.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:53:37.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776130 2024-06-15T10:53:37.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:37.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-06-15T10:53:37.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-06-15T10:53:37.798 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 292057776130 2024-06-15T10:53:37.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:53:38.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776130 -lt 292057776130 2024-06-15T10:53:38.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:53:38.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:38.042 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:38.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:53:38.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:53:38.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:53:38.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:53:38.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:53:38.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:38.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:38.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:53:38.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:53:38.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:53:38.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:38.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:38.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:53:38.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:53:38.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:53:38.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:53:38.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:53:38.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:53:38.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:53:38.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:53:38.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:53:38.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:53:39.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:53:39.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:40.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:53:40.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:53:40.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:53:40.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:53:40.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:53:40.880 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:53:40.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:53:40.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:53:40.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:53:40.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:53:40.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:53:40.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:53:40.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:53:40.883 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:53:40.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:53:40.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:53:40.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:53:40.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:53:40.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:53:40.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:53:40.917 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:40.918+0000 7fa1340dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:40.920 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:40.922+0000 7fa1340dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:40.921 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:40.922+0000 7fa1340dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:41.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:41.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:53:41.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:41.457 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:41.458+0000 7fa1340dd7c0 -1 Falling back to public interface 2024-06-15T10:53:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:42.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:53:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:53:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:42.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:53:42.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:53:43.620 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:53:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:53:43.742 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:43.742+0000 7fa1340dd7c0 -1 osd.0 70 log_to_monitors true 2024-06-15T10:53:43.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:44.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:53:44.854 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:53:44.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:44.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:53:45.117 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/1014635362,v1:127.0.0.1:6803/1014635362] [v2:127.0.0.1:6804/1014635362,v1:127.0.0.1:6805/1014635362] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:53:45.118 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:53:45.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:53:45.119 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:53:45.119 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:53:45.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:53:45.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:53:45.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:53:45.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:53:45.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:53:45.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:53:45.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:53:45.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:53:45.225 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:53:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:53:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:53:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:53:45.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:45.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:53:45.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-06-15T10:53:45.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-06-15T10:53:45.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906' 2024-06-15T10:53:45.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:45.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:53:45.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776133 2024-06-15T10:53:45.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776133 2024-06-15T10:53:45.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-06-15T10:53:45.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:45.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579906 2024-06-15T10:53:45.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:45.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:53:45.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579906 2024-06-15T10:53:45.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-06-15T10:53:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-06-15T10:53:45.751 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 317827579906 2024-06-15T10:53:45.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:45.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 317827579906 2024-06-15T10:53:45.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:53:46.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:53:46.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:47.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-06-15T10:53:47.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:47.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776133 2024-06-15T10:53:47.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:47.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:53:47.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776133 2024-06-15T10:53:47.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776133 2024-06-15T10:53:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-06-15T10:53:47.222 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 292057776133 2024-06-15T10:53:47.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:53:47.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776133 -lt 292057776133 2024-06-15T10:53:47.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:53:47.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:47.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:47.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:53:47.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:53:47.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:53:47.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:53:47.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:53:47.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:53:47.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:53:47.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:53:48.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:53:48.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:53:48.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:48.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:48.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:53:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:53:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:53:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:48.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-06-15T10:53:48.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-06-15T10:53:48.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:53:48.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:53:48.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:53:48.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-06-15T10:53:49.884 INFO:tasks.workunit.client.0.smithi116.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:53:50.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:53:50.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:53:50.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:53:50.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:53:50.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:53:50.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:53:50.433 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:53:50.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:53:50.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:53:50.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:53:50.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:53:50.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:53:50.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:53:50.464 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:50.466+0000 7f1d4e9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:50.468 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:50.470+0000 7f1d4e9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:50.469 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:50.470+0000 7f1d4e9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:53:50.685 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:53:50.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:50.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:50.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:51.010 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:51.010+0000 7f1d4e9047c0 -1 Falling back to public interface 2024-06-15T10:53:51.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:53:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:53:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:51.926 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:52.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:53:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:53:53.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:53.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:53.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:53:53.434 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:53:53.434+0000 7f1d4e9047c0 -1 osd.1 75 log_to_monitors true 2024-06-15T10:53:54.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:53:54.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:53:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:53:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:53:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:53:54.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/246395630,v1:127.0.0.1:6811/246395630] [v2:127.0.0.1:6812/246395630,v1:127.0.0.1:6813/246395630] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:53:54.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:53:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:53:54.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:53:54.651 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:53:54.652 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:53:54.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:53:54.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:53:54.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:53:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:53:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:53:54.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:53:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:53:54.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:53:54.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:53:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:53:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:53:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:53:55.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:55.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:53:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579909 2024-06-15T10:53:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579909 2024-06-15T10:53:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909' 2024-06-15T10:53:55.171 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:53:55.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:53:55.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-06-15T10:53:55.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-06-15T10:53:55.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-06-15T10:53:55.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:55.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579909 2024-06-15T10:53:55.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:55.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:53:55.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579909 2024-06-15T10:53:55.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:55.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579909 2024-06-15T10:53:55.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-06-15T10:53:55.321 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 317827579909 2024-06-15T10:53:55.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:55.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579908 -lt 317827579909 2024-06-15T10:53:55.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:53:56.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:53:56.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:53:56.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579909 -lt 317827579909 2024-06-15T10:53:56.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:53:56.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416386 2024-06-15T10:53:56.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:53:56.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:53:56.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416386 2024-06-15T10:53:56.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:53:56.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-06-15T10:53:56.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-06-15T10:53:56.854 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 339302416386 2024-06-15T10:53:56.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:53:57.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416386 -lt 339302416386 2024-06-15T10:53:57.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:53:57.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:57.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:57.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:53:57.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:53:57.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:53:57.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:53:57.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:53:57.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:53:57.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:53:57.378 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:53:57.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:53:57.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:53:57.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:53:57.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:53:57.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:53:57.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:53:57.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:53:57.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:53:57.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-06-15T10:53:57.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-06-15T10:53:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:53:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:53:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:57.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:53:57.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:53:57.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:53:57.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:53:57.867 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:53:57.867 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:53:57.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:53:57.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:53:58.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:53:58.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-06-15T10:53:59.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:53:59.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:53:59.978 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:53:59.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:53:59.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:53:59.981 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:53:59.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:53:59.981 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:53:59.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:53:59.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:53:59.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:53:59.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:00.027 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:00.026+0000 7fe2c23c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:00.030 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:00.030+0000 7fe2c23c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:00.031 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:00.034+0000 7fe2c23c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:00.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:00.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:00.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:00.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:00.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:00.586+0000 7fe2c23c27c0 -1 Falling back to public interface 2024-06-15T10:54:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:01.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:01.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:01.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:02.731 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:02.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:02.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:02.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:03.258+0000 7fe2c23c27c0 -1 osd.0 80 log_to_monitors true 2024-06-15T10:54:03.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:03.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:03.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:03.972 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:03.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:03.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:04.280 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/60684295,v1:127.0.0.1:6803/60684295] [v2:127.0.0.1:6804/60684295,v1:127.0.0.1:6805/60684295] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:54:04.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:04.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:04.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:04.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:04.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:04.281 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:04.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:04.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:04.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:04.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:04.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:04.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:04.394 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:04.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:04.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:04.627 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:04.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:04.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:04.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-06-15T10:54:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-06-15T10:54:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866' 2024-06-15T10:54:04.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:04.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416389 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416389 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252866 2024-06-15T10:54:04.920 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:04.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:04.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252866 2024-06-15T10:54:04.923 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:04.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-06-15T10:54:04.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-06-15T10:54:04.925 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 360777252866 2024-06-15T10:54:04.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:05.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-06-15T10:54:05.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:06.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:06.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:06.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-06-15T10:54:06.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:07.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:54:07.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:07.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-06-15T10:54:07.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:07.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:07.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416389 2024-06-15T10:54:07.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:07.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416389 2024-06-15T10:54:07.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:07.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416389 2024-06-15T10:54:07.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-06-15T10:54:07.694 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 339302416389 2024-06-15T10:54:07.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:07.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416389 -lt 339302416389 2024-06-15T10:54:07.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:07.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:07.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:08.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:08.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:08.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:08.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:08.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:08.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:08.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-06-15T10:54:08.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:54:08.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:08.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:54:08.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:08.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:08.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:08.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:08.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:09.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:09.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:09.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:10.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:54:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:54:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:10.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:10.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:10.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:10.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:10.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:10.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:10.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:54:10.882 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:54:10.883 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:54:10.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:10.883 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:54:10.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:54:10.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:10.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:10.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:10.916 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:10.918+0000 7fab790ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:10.919 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:10.918+0000 7fab790ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:10.920 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:10.922+0000 7fab790ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:54:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:54:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:11.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:11.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:11.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:11.450 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:11.449+0000 7fab790ae7c0 -1 Falling back to public interface 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:12.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:12.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:13.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:13.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:13.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:13.618 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:13.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:13.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:13.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:14.146 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:14.145+0000 7fab790ae7c0 -1 osd.1 86 log_to_monitors true 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:14.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/1983970783,v1:127.0.0.1:6811/1983970783] [v2:127.0.0.1:6812/1983970783,v1:127.0.0.1:6813/1983970783] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:15.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:15.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:15.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:15.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:15.334 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:15.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:15.627 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:15.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:15.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:15.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:15.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252869 2024-06-15T10:54:15.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252869 2024-06-15T10:54:15.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869' 2024-06-15T10:54:15.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:15.789 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-06-15T10:54:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-06-15T10:54:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-06-15T10:54:15.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:15.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252869 2024-06-15T10:54:15.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:15.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:15.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252869 2024-06-15T10:54:15.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252869 2024-06-15T10:54:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-06-15T10:54:15.934 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 360777252869 2024-06-15T10:54:15.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:16.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252868 -lt 360777252869 2024-06-15T10:54:16.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:17.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:17.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252869 -lt 360777252869 2024-06-15T10:54:17.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:17.409 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-06-15T10:54:17.410 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:17.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:17.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-06-15T10:54:17.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-06-15T10:54:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-06-15T10:54:17.413 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 386547056642 2024-06-15T10:54:17.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:17.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-06-15T10:54:17.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:17.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:17.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:17.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:17.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:17.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:18.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:18.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:18.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:18.117 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:18.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:18.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:18.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:18.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:54:18.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-06-15T10:54:18.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-06-15T10:54:18.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:18.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:18.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:18.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:18.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:18.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:54:18.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-06-15T10:54:20.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:54:20.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:20.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:20.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:20.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:20.352 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:20.352 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:20.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:54:20.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:54:20.355 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:54:20.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:20.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:54:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:54:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:20.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:20.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:20.389 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:20.389+0000 7f6ce7adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:20.392 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:20.393+0000 7f6ce7adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:20.393 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:20.393+0000 7f6ce7adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:20.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:20.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:20.962 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:20.961+0000 7f6ce7adb7c0 -1 Falling back to public interface 2024-06-15T10:54:21.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:21.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:21.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:21.872 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:21.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:21.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:22.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:23.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:23.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:23.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:23.114 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:23.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:23.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:23.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:23.369 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:23.369+0000 7f6ce7adb7c0 -1 osd.0 91 log_to_monitors true 2024-06-15T10:54:24.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:24.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:24.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:24.358 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:24.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:24.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/177754759,v1:127.0.0.1:6803/177754759] [v2:127.0.0.1:6804/177754759,v1:127.0.0.1:6805/177754759] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:24.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:24.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:24.653 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:24.653 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:24.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:24.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:24.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:24.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:24.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:25.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:25.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:25.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:25.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-06-15T10:54:25.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-06-15T10:54:25.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122' 2024-06-15T10:54:25.193 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:25.194 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:25.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-06-15T10:54:25.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-06-15T10:54:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122 1-386547056645' 2024-06-15T10:54:25.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:25.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893122 2024-06-15T10:54:25.338 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:25.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:25.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893122 2024-06-15T10:54:25.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:25.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-06-15T10:54:25.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-06-15T10:54:25.341 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 408021893122 2024-06-15T10:54:25.342 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:25.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 408021893122 2024-06-15T10:54:25.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:26.571 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:26.572 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:26.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-06-15T10:54:26.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:26.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-06-15T10:54:26.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:26.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:26.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-06-15T10:54:26.844 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:26.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-06-15T10:54:26.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-06-15T10:54:26.845 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 386547056645 2024-06-15T10:54:26.846 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:27.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056645 -lt 386547056645 2024-06-15T10:54:27.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:27.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:27.104 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:27.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:27.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:27.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:27.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:27.382 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:27.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:27.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:27.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:27.626 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:27.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:27.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:27.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:27.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:27.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:27.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-06-15T10:54:27.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:54:27.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-06-15T10:54:27.886 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-06-15T10:54:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:54:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:54:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-06-15T10:54:27.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-15T10:54:27.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:27.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:27.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:27.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:27.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:28.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:28.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-15T10:54:28.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:28.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:28.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:54:28.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:28.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:28.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:29.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:29.836 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:29.837 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:29.837 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:29.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:54:29.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:29.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:54:29.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:54:29.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:29.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:29.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:29.877 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:29.877+0000 7fa89f7147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:29.880 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:29.881+0000 7fa89f7147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:29.882 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:29.881+0000 7fa89f7147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:30.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:30.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:30.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:30.422 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:30.421+0000 7fa89f7147c0 -1 Falling back to public interface 2024-06-15T10:54:31.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:31.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:31.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:31.340 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:31.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:31.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:31.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:32.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:32.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:32.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:32.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:32.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:33.000 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:33.001+0000 7fa89f7147c0 -1 osd.1 97 log_to_monitors true 2024-06-15T10:54:33.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:33.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:33.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:33.813 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:33.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:33.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:34.105 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/2513890761,v1:127.0.0.1:6811/2513890761] [v2:127.0.0.1:6812/2513890761,v1:127.0.0.1:6813/2513890761] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:54:34.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:34.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:34.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:34.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:34.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:34.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:34.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:34.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:34.627 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893125 2024-06-15T10:54:34.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893125 2024-06-15T10:54:34.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125' 2024-06-15T10:54:34.628 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:34.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:34.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-06-15T10:54:34.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-06-15T10:54:34.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125 1-433791696898' 2024-06-15T10:54:34.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:34.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893125 2024-06-15T10:54:34.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:34.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:34.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893125 2024-06-15T10:54:34.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:34.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893125 2024-06-15T10:54:34.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893125' 2024-06-15T10:54:34.773 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 408021893125 2024-06-15T10:54:34.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:34.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893125 2024-06-15T10:54:34.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:35.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:35.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:36.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893125 2024-06-15T10:54:36.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:37.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:54:37.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:37.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893125 -lt 408021893125 2024-06-15T10:54:37.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:37.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696898 2024-06-15T10:54:37.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:37.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:37.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696898 2024-06-15T10:54:37.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:37.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-06-15T10:54:37.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-06-15T10:54:37.493 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 433791696898 2024-06-15T10:54:37.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:37.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-06-15T10:54:37.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:37.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:37.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:37.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:37.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:38.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:38.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:38.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:38.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:38.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:38.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:38.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:38.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:38.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-06-15T10:54:38.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:54:38.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-06-15T10:54:38.485 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-06-15T10:54:38.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:54:38.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:54:38.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-06-15T10:54:38.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:54:38.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:38.778 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:38.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:38.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:38.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:39.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:39.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-15T10:54:39.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:39.090 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:39.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:54:39.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:39.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:54:39.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-06-15T10:54:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:54:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:40.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:54:40.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:54:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:40.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:40.544 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:40.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:54:40.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:54:40.546 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:54:40.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:40.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:54:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:54:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:40.579 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:40.577+0000 7ffaed31b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:40.581 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:40.581+0000 7ffaed31b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:40.583 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:40.581+0000 7ffaed31b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:40.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:40.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:41.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:41.118 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:41.117+0000 7ffaed31b7c0 -1 Falling back to public interface 2024-06-15T10:54:42.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:42.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:42.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:42.039 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:42.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:42.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:42.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:43.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:43.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:43.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:43.519 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:43.521+0000 7ffaed31b7c0 -1 osd.0 102 log_to_monitors true 2024-06-15T10:54:44.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:44.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:54:44.908 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/3014180670,v1:127.0.0.1:6803/3014180670] [v2:127.0.0.1:6804/3014180670,v1:127.0.0.1:6805/3014180670] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:54:44.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:44.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:44.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:44.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:44.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:44.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:44.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:44.910 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:45.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:45.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:45.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:45.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:45.264 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:45.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:45.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:45.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-06-15T10:54:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-06-15T10:54:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378' 2024-06-15T10:54:45.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:45.412 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-06-15T10:54:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-06-15T10:54:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378 1-433791696901' 2024-06-15T10:54:45.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533378 2024-06-15T10:54:45.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:45.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533378 2024-06-15T10:54:45.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-06-15T10:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-06-15T10:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 455266533378 2024-06-15T10:54:45.557 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:45.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 455266533378 2024-06-15T10:54:45.784 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:46.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:46.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:47.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-06-15T10:54:47.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:47.028 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:47.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696901 2024-06-15T10:54:47.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:47.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696901 2024-06-15T10:54:47.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:47.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-06-15T10:54:47.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696901' 2024-06-15T10:54:47.032 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 433791696901 2024-06-15T10:54:47.033 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:47.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-06-15T10:54:47.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:47.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:47.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:47.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:47.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:47.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:47.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:47.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:47.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:47.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:48.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:48.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:48.092 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:54:48.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:48.094 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:50.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:50.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:50.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:50.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:54:50.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:54:50.302 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:54:50.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:50.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:54:50.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:54:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:50.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:50.334 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:50.333+0000 7fcb3bca67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:50.337 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:50.337+0000 7fcb3bca67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:50.338 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:50.337+0000 7fcb3bca67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:50.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:54:50.548 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:54:50.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:50.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:50.905 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:50.905+0000 7fcb3bca67c0 -1 Falling back to public interface 2024-06-15T10:54:51.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:51.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:51.843 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:54:51.844 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:54:51.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:51.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:52.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:53.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:53.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:54:53.504 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:53.505+0000 7fcb3bca67c0 -1 osd.1 108 log_to_monitors true 2024-06-15T10:54:54.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:54:54.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:54:54.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:54:54.330 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:54:54.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:54:54.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/1373663051,v1:127.0.0.1:6811/1373663051] [v2:127.0.0.1:6812/1373663051,v1:127.0.0.1:6813/1373663051] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:54:54.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:54:54.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:54:54.673 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:54:54.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:54:54.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:54:54.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:54:54.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:54:54.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:54:54.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:54:54.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:54:54.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:54:54.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:54:54.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:54:55.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:54:55.089 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:54:55.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:54:55.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:55.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:54:55.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-06-15T10:54:55.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-06-15T10:54:55.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381' 2024-06-15T10:54:55.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:54:55.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:54:55.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-06-15T10:54:55.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-06-15T10:54:55.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381 1-481036337154' 2024-06-15T10:54:55.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:55.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533381 2024-06-15T10:54:55.396 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:55.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:54:55.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533381 2024-06-15T10:54:55.398 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:55.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-06-15T10:54:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533381' 2024-06-15T10:54:55.400 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 455266533381 2024-06-15T10:54:55.400 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:55.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533381 2024-06-15T10:54:55.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:54:56.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:54:56.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:54:56.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533381 2024-06-15T10:54:56.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:54:56.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-481036337154 2024-06-15T10:54:56.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:54:56.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:54:56.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-481036337154 2024-06-15T10:54:56.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:54:56.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-06-15T10:54:56.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-06-15T10:54:56.909 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 481036337154 2024-06-15T10:54:56.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:54:57.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-06-15T10:54:57.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:54:57.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:57.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:57.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:54:57.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:54:57.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:54:57.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:54:57.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:54:57.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:54:57.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:54:57.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:54:57.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:54:57.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:54:57.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:54:57.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:54:57.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:54:57.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:54:57.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:54:58.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:54:58.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-15T10:54:58.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:54:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:54:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:54:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:54:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:58.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:54:59.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:54:59.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:54:59.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:54:59.897 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:54:59.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:54:59.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:54:59.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:54:59.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:54:59.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:54:59.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:54:59.931 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:59.929+0000 7f954e2db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:59.934 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:59.933+0000 7f954e2db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:54:59.936 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:54:59.937+0000 7f954e2db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:00.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:00.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:00.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:00.502 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:00.501+0000 7f954e2db7c0 -1 Falling back to public interface 2024-06-15T10:55:01.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:01.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:01.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:01.427 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:01.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:01.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:01.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:02.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:02.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:02.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:02.692 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:02.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:02.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:03.059 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:03.057+0000 7f954e2db7c0 -1 osd.1 113 log_to_monitors true 2024-06-15T10:55:03.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:03.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:03.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:03.934 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:03.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:03.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:04.323 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/114511505,v1:127.0.0.1:6811/114511505] [v2:127.0.0.1:6812/114511505,v1:127.0.0.1:6813/114511505] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:04.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:04.325 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:04.325 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:04.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:04.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:04.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:04.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:04.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:04.774 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:04.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:04.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:04.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:05.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533384 2024-06-15T10:55:05.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533384 2024-06-15T10:55:05.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533384' 2024-06-15T10:55:05.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:05.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-06-15T10:55:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-06-15T10:55:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533384 1-502511173634' 2024-06-15T10:55:05.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:05.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533384 2024-06-15T10:55:05.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:05.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:05.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533384 2024-06-15T10:55:05.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:05.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533384 2024-06-15T10:55:05.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533384' 2024-06-15T10:55:05.156 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 455266533384 2024-06-15T10:55:05.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:05.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533383 -lt 455266533384 2024-06-15T10:55:05.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:06.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:55:06.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:06.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533383 -lt 455266533384 2024-06-15T10:55:06.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:07.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:55:07.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:07.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533384 -lt 455266533384 2024-06-15T10:55:07.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:07.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-502511173634 2024-06-15T10:55:07.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:07.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:07.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-502511173634 2024-06-15T10:55:07.857 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:07.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-06-15T10:55:07.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-06-15T10:55:07.859 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 502511173634 2024-06-15T10:55:07.859 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:55:08.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-06-15T10:55:08.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:55:08.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:08.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:08.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:55:08.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:08.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:55:08.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:08.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:08.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:55:08.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:55:08.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:55:08.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:55:08.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:55:08.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:55:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:55:08.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:08.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:08.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:08.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:08.950 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:08.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:08.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:09.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:55:09.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:09.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:09.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:55:09.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:55:09.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:55:09.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:55:09.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:55:09.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:55:09.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:55:09.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:55:09.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:55:09.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:55:09.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:09.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:09.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:09.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:09.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:09.876 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:09.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:10.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:55:10.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:10.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:10.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:10.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:55:10.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:55:10.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:55:10.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:55:10.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:55:10.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:55:10.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:55:10.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:55:10.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:55:10.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:55:10.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:55:10.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:55:10.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:55:10.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:55:11.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:55:11.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:11.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:11.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:11.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:11.060 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:11.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:11.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:11.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:55:11.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:11.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:11.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:11.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:55:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:55:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:55:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-06-15T10:55:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:11.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-15T10:55:11.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-06-15T10:55:11.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:55:11.547 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:55:11.548 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:55:11.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:55:11.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:55:11.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:11.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-06-15T10:55:13.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:55:13.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:55:13.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:55:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:55:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:55:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:55:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:55:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:55:13.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:55:13.469 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:55:13.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:55:13.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:55:13.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:55:13.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:55:13.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:55:13.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:55:13.504 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:13.504+0000 7f8c93ac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:13.507 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:13.504+0000 7f8c93ac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:13.509 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:13.508+0000 7f8c93ac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:13.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:55:13.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:13.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:13.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:13.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:14.046 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:14.044+0000 7f8c93ac37c0 -1 Falling back to public interface 2024-06-15T10:55:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:14.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:14.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:15.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:16.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:16.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:16.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:16.191 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:16.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:16.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:16.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:16.504 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:16.504+0000 7f8c93ac37c0 -1 osd.1 119 log_to_monitors true 2024-06-15T10:55:17.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:17.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:17.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:17.431 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:17.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:17.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:17.854 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 123 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/2906096642,v1:127.0.0.1:6811/2906096642] [v2:127.0.0.1:6812/2906096642,v1:127.0.0.1:6813/2906096642] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:55:17.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:17.855 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:17.856 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:17.856 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:17.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:17.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:17.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:17.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:17.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:17.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:17.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:17.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:17.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:18.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:18.272 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:18.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:18.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:18.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:18.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533387 2024-06-15T10:55:18.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533387 2024-06-15T10:55:18.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387' 2024-06-15T10:55:18.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:18.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:18.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-06-15T10:55:18.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-06-15T10:55:18.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387 1-528280977410' 2024-06-15T10:55:18.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:18.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533387 2024-06-15T10:55:18.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:18.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:18.575 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533387 2024-06-15T10:55:18.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533387 2024-06-15T10:55:18.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533387' 2024-06-15T10:55:18.578 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 455266533387 2024-06-15T10:55:18.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:18.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533387 -lt 455266533387 2024-06-15T10:55:18.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:18.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-528280977410 2024-06-15T10:55:18.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:18.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:18.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-528280977410 2024-06-15T10:55:18.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-06-15T10:55:18.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-06-15T10:55:18.865 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 528280977410 2024-06-15T10:55:18.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:55:19.139 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-06-15T10:55:19.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:55:19.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:19.140 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:19.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:55:19.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:19.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:19.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:55:19.653 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:19.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:19.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:19.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:55:19.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:55:19.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:55:19.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-06-15T10:55:19.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:55:20.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:55:20.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:55:20.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:55:20.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:55:20.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:55:20.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:55:20.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:55:20.378 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:55:20.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:20.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:55:22.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:55:22.413 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:55:22.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:55:22.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:55:22.414 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:55:22.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:55:22.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:55:22.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:55:22.417 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:55:22.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:55:22.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:55:22.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:55:22.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:55:22.454 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:22.452+0000 7f0f344137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:22.457 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:22.456+0000 7f0f344137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:22.459 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:22.456+0000 7f0f344137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:22.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:22.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:22.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:22.994 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:22.992+0000 7f0f344137c0 -1 Falling back to public interface 2024-06-15T10:55:23.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:23.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:23.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:23.897 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:23.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:23.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:24.135 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:25.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:25.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:25.428 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:25.428+0000 7f0f344137c0 -1 osd.1 125 log_to_monitors true 2024-06-15T10:55:26.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:26.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:26.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:26.382 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:26.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:26.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 129 up_thru 129 down_at 126 last_clean_interval [123,125) [v2:127.0.0.1:6810/108836076,v1:127.0.0.1:6811/108836076] [v2:127.0.0.1:6812/108836076,v1:127.0.0.1:6813/108836076] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:26.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:26.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:26.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:26.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:26.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:26.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:26.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:26.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:27.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:27.020 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:27.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:27.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:27.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:27.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533390 2024-06-15T10:55:27.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533390 2024-06-15T10:55:27.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390' 2024-06-15T10:55:27.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:27.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:27.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781186 2024-06-15T10:55:27.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781186 2024-06-15T10:55:27.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390 1-554050781186' 2024-06-15T10:55:27.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:27.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533390 2024-06-15T10:55:27.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:27.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:27.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533390 2024-06-15T10:55:27.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:27.328 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533390 2024-06-15T10:55:27.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533390' 2024-06-15T10:55:27.329 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 455266533390 2024-06-15T10:55:27.329 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:27.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533389 -lt 455266533390 2024-06-15T10:55:27.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:28.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:55:28.584 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:28.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533390 -lt 455266533390 2024-06-15T10:55:28.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:28.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781186 2024-06-15T10:55:28.871 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:28.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:28.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781186 2024-06-15T10:55:28.874 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:28.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781186 2024-06-15T10:55:28.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781186' 2024-06-15T10:55:28.875 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 554050781186 2024-06-15T10:55:28.875 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:55:29.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781186 -lt 554050781186 2024-06-15T10:55:29.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:55:29.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:29.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:55:29.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:29.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:29.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:29.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:55:29.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:29.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:29.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:29.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:55:29.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:55:29.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:55:29.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-15T10:55:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-15T10:55:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:29.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-15T10:55:29.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-06-15T10:55:29.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:55:29.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:29.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:29.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-06-15T10:55:29.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-06-15T10:55:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:55:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:29.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-06-15T10:55:29.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-06-15T10:55:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:55:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:29.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-06-15T10:55:29.913 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-06-15T10:55:29.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:55:29.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:29.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:29.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-06-15T10:55:29.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-06-15T10:55:29.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:55:29.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:29.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-15T10:55:29.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:55:29.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:55:29.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:55:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:55:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:55:29.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:55:30.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-06-15T10:55:31.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:55:31.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:55:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:55:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:55:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:55:31.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:55:31.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:55:31.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:55:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:55:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:55:31.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:55:31.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:55:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:55:31.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:55:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:55:31.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:55:31.942 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:31.940+0000 7fa7d672d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:31.945 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:31.944+0000 7fa7d672d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:31.947 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:31.944+0000 7fa7d672d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:32.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:55:32.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:32.481 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:32.480+0000 7fa7d672d7c0 -1 Falling back to public interface 2024-06-15T10:55:33.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:33.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:33.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:33.405 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:33.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:33.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:55:33.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:34.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:34.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:55:34.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:35.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:35.236+0000 7fa7d672d7c0 -1 osd.0 131 log_to_monitors true 2024-06-15T10:55:35.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:35.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:35.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:35.909 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:35.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:35.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 135 up_thru 135 down_at 132 last_clean_interval [106,131) [v2:127.0.0.1:6802/2945824942,v1:127.0.0.1:6803/2945824942] [v2:127.0.0.1:6804/2945824942,v1:127.0.0.1:6805/2945824942] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:36.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:36.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:36.319 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:36.320 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:36.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:36.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:36.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:36.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:36.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:36.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:36.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:36.681 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:36.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:36.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:36.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:36.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584962 2024-06-15T10:55:36.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584962 2024-06-15T10:55:36.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962' 2024-06-15T10:55:36.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:36.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781189 2024-06-15T10:55:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781189 2024-06-15T10:55:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962 1-554050781189' 2024-06-15T10:55:36.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:36.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584962 2024-06-15T10:55:36.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:36.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:36.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584962 2024-06-15T10:55:36.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:36.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584962 2024-06-15T10:55:36.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584962' 2024-06-15T10:55:36.976 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 579820584962 2024-06-15T10:55:36.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:37.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 579820584962 2024-06-15T10:55:37.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:38.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:55:38.201 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 579820584962 2024-06-15T10:55:38.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:39.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:55:39.440 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:39.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584962 -lt 579820584962 2024-06-15T10:55:39.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:39.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781189 2024-06-15T10:55:39.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:39.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:39.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781189 2024-06-15T10:55:39.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781189 2024-06-15T10:55:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781189' 2024-06-15T10:55:39.697 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 554050781189 2024-06-15T10:55:39.697 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:55:39.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781189 -lt 554050781189 2024-06-15T10:55:39.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:55:39.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:39.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:40.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:55:40.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:40.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:40.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:55:40.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:40.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:40.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:40.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:55:40.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:55:40.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:55:40.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:55:40.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:55:40.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:55:40.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:55:41.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:55:41.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:41.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:42.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:55:42.744 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:42.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:55:42.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:55:42.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:55:42.748 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:55:42.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:55:42.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:55:42.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:55:42.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:55:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:55:42.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:55:42.781 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:42.780+0000 7f72bc96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:42.785 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:42.784+0000 7f72bc96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:42.786 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:42.784+0000 7f72bc96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:42.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:55:42.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:42.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:55:42.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:42.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:43.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:43.320+0000 7f72bc96a7c0 -1 Falling back to public interface 2024-06-15T10:55:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:44.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:44.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:44.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:45.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:45.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:45.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:45.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:45.945 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:45.944+0000 7f72bc96a7c0 -1 osd.1 136 log_to_monitors true 2024-06-15T10:55:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:46.752 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:46.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:47.120 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 140 up_thru 140 down_at 137 last_clean_interval [129,136) [v2:127.0.0.1:6810/3096918818,v1:127.0.0.1:6811/3096918818] [v2:127.0.0.1:6812/3096918818,v1:127.0.0.1:6813/3096918818] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:47.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:47.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:47.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:47.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:47.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:47.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:47.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:47.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:47.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:47.513 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:47.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:47.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:47.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:47.660 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584965 2024-06-15T10:55:47.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584965 2024-06-15T10:55:47.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965' 2024-06-15T10:55:47.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:47.661 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:47.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=601295421442 2024-06-15T10:55:47.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 601295421442 2024-06-15T10:55:47.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965 1-601295421442' 2024-06-15T10:55:47.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:47.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584965 2024-06-15T10:55:47.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:47.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:47.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584965 2024-06-15T10:55:47.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:47.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584965 2024-06-15T10:55:47.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584965' 2024-06-15T10:55:47.802 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 579820584965 2024-06-15T10:55:47.803 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:48.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584964 -lt 579820584965 2024-06-15T10:55:48.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:49.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:55:49.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:49.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584965 -lt 579820584965 2024-06-15T10:55:49.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:49.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-601295421442 2024-06-15T10:55:49.245 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:49.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:49.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-601295421442 2024-06-15T10:55:49.248 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=601295421442 2024-06-15T10:55:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2024-06-15T10:55:49.250 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 601295421442 2024-06-15T10:55:49.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:55:49.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 601295421442 -lt 601295421442 2024-06-15T10:55:49.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:55:49.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:49.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:49.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:55:49.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:55:49.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:55:50.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:55:50.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:55:50.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:55:50.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:55:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:55:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:55:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:55:50.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-06-15T10:55:50.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:55:50.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-06-15T10:55:50.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:55:50.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:55:50.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:55:50.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:50.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-06-15T10:55:51.698 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:55:52.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:55:52.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:55:52.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:55:52.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:55:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:55:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:55:52.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:55:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:55:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:55:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:55:52.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:52.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:55:52.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:55:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:55:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:55:52.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:55:52.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:55:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:55:52.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:55:52.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:55:52.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:55:52.270 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:52.268+0000 7f28a9bf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:52.272 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:52.272+0000 7f28a9bf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:52.274 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:52.272+0000 7f28a9bf47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:55:52.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:55:52.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:55:52.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:52.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:52.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:52.820+0000 7f28a9bf47c0 -1 Falling back to public interface 2024-06-15T10:55:53.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:55:53.742 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:55:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:53.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:53.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:54.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:54.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:54.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:55:54.984 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:55:54.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:54.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:55.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:55:55.245 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:55:55.244+0000 7f28a9bf47c0 -1 osd.1 142 log_to_monitors true 2024-06-15T10:55:56.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:55:56.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:55:56.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:55:56.233 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:55:56.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:55:56.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:55:56.517 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 146 up_thru 146 down_at 143 last_clean_interval [140,142) [v2:127.0.0.1:6810/2604498700,v1:127.0.0.1:6811/2604498700] [v2:127.0.0.1:6812/2604498700,v1:127.0.0.1:6813/2604498700] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:55:56.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:55:56.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:55:56.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:55:56.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:55:56.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:55:56.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:55:56.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:55:56.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:55:56.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:55:56.521 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:55:56.521 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:55:56.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:55:56.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:55:56.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:55:56.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:55:56.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:55:56.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:55:56.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:55:56.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:55:56.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:56.894 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:55:57.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584968 2024-06-15T10:55:57.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584968 2024-06-15T10:55:57.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584968' 2024-06-15T10:55:57.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:55:57.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:55:57.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225218 2024-06-15T10:55:57.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225218 2024-06-15T10:55:57.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584968 1-627065225218' 2024-06-15T10:55:57.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:57.187 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584968 2024-06-15T10:55:57.188 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:57.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:55:57.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584968 2024-06-15T10:55:57.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:57.190 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584968 2024-06-15T10:55:57.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584968' 2024-06-15T10:55:57.191 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 579820584968 2024-06-15T10:55:57.191 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:57.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584967 -lt 579820584968 2024-06-15T10:55:57.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:58.422 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:55:58.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:58.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584967 -lt 579820584968 2024-06-15T10:55:58.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:55:59.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:55:59.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:55:59.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584968 -lt 579820584968 2024-06-15T10:55:59.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:55:59.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:55:59.912 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225218 2024-06-15T10:55:59.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:55:59.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225218 2024-06-15T10:55:59.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:55:59.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225218 2024-06-15T10:55:59.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225218' 2024-06-15T10:55:59.916 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 627065225218 2024-06-15T10:55:59.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:00.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225218 -lt 627065225218 2024-06-15T10:56:00.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:00.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:00.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:00.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:00.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:00.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:00.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:00.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:00.718 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:00.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:00.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:00.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:00.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:00.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:56:00.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-06-15T10:56:00.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:00.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-15T10:56:00.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:00.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:00.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:56:00.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:00.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:56:00.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:00.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:00.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:00.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:00.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:01.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-06-15T10:56:02.466 INFO:tasks.workunit.client.0.smithi116.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:03.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:03.259 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:03.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:56:03.262 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:56:03.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:03.263 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:56:03.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:56:03.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:03.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:03.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:03.296 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:03.296+0000 7fd15256e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:03.298 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:03.296+0000 7fd15256e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:03.300 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:03.300+0000 7fd15256e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:03.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:03.506 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:03.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:03.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:03.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:03.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:03.836+0000 7fd15256e7c0 -1 Falling back to public interface 2024-06-15T10:56:04.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:04.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:04.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:04.759 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:04.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:04.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:04.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:56:06.000 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:56:06.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:06.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:06.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:06.363 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:06.360+0000 7fd15256e7c0 -1 osd.0 148 log_to_monitors true 2024-06-15T10:56:07.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:07.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:07.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:56:07.245 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:56:07.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:07.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 152 up_thru 152 down_at 149 last_clean_interval [135,148) [v2:127.0.0.1:6802/186997651,v1:127.0.0.1:6803/186997651] [v2:127.0.0.1:6804/186997651,v1:127.0.0.1:6805/186997651] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:56:07.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:56:07.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:56:07.496 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:56:07.497 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:56:07.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:56:07.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:56:07.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:56:07.618 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:56:07.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:56:07.886 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:56:07.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:56:07.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:07.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:56:08.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028994 2024-06-15T10:56:08.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028994 2024-06-15T10:56:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994' 2024-06-15T10:56:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:08.124 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:56:08.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225221 2024-06-15T10:56:08.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225221 2024-06-15T10:56:08.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994 1-627065225221' 2024-06-15T10:56:08.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:08.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028994 2024-06-15T10:56:08.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:08.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:56:08.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028994 2024-06-15T10:56:08.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:08.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028994 2024-06-15T10:56:08.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028994' 2024-06-15T10:56:08.273 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 652835028994 2024-06-15T10:56:08.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:08.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584969 -lt 652835028994 2024-06-15T10:56:08.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:09.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:56:09.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:09.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028994 -lt 652835028994 2024-06-15T10:56:09.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:09.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225221 2024-06-15T10:56:09.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:09.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:56:09.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225221 2024-06-15T10:56:09.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:09.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225221 2024-06-15T10:56:09.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225221' 2024-06-15T10:56:09.779 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 627065225221 2024-06-15T10:56:09.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:10.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225221 -lt 627065225221 2024-06-15T10:56:10.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:10.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:10.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:10.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:10.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:10.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:10.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:10.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:10.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:10.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:10.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:10.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:10.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:10.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:10.518 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:10.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:10.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:10.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:10.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-06-15T10:56:10.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:10.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:10.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:12.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:12.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-06-15T10:56:13.168 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:56:13.168 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:13.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:13.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:56:13.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:56:13.706 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:56:13.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:13.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:56:13.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:56:13.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:13.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:13.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:13.736+0000 7ff2d052b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:13.742 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:13.740+0000 7ff2d052b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:13.744 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:13.744+0000 7ff2d052b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:13.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:56:13.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:13.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:14.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:14.294 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:14.291+0000 7ff2d052b7c0 -1 Falling back to public interface 2024-06-15T10:56:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:15.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:15.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:15.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:16.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:16.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:16.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:56:16.426 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:56:16.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:16.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:16.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:16.867 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:16.863+0000 7ff2d052b7c0 -1 osd.1 154 log_to_monitors true 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:17.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:17.913 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 158 up_thru 158 down_at 155 last_clean_interval [146,154) [v2:127.0.0.1:6810/2172186334,v1:127.0.0.1:6811/2172186334] [v2:127.0.0.1:6812/2172186334,v1:127.0.0.1:6813/2172186334] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:56:17.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:56:17.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:56:17.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:56:17.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:56:17.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:56:17.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:56:17.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:56:17.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:56:17.915 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:56:17.915 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:56:17.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:56:17.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:56:17.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:56:18.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:56:18.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:56:18.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:56:18.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:56:18.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:56:18.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:56:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:56:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:56:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:56:18.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:18.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:56:18.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028997 2024-06-15T10:56:18.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028997 2024-06-15T10:56:18.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997' 2024-06-15T10:56:18.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:18.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:56:18.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832770 2024-06-15T10:56:18.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832770 2024-06-15T10:56:18.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997 1-678604832770' 2024-06-15T10:56:18.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:18.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028997 2024-06-15T10:56:18.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:18.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:56:18.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028997 2024-06-15T10:56:18.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:18.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028997 2024-06-15T10:56:18.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028997' 2024-06-15T10:56:18.712 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 652835028997 2024-06-15T10:56:18.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:18.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028997 -lt 652835028997 2024-06-15T10:56:18.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:18.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832770 2024-06-15T10:56:18.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:18.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:56:18.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832770 2024-06-15T10:56:18.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:18.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832770 2024-06-15T10:56:18.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832770' 2024-06-15T10:56:18.947 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 678604832770 2024-06-15T10:56:18.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:19.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 678604832770 2024-06-15T10:56:19.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:20.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:56:20.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:20.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 678604832770 2024-06-15T10:56:20.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:21.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:56:21.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:21.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832770 -lt 678604832770 2024-06-15T10:56:21.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:21.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:21.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:21.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:21.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:21.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:21.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:21.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:21.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:21.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:21.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:22.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:22.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:22.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:22.217 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:22.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:22.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:22.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:22.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:56:22.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-06-15T10:56:22.474 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-06-15T10:56:22.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:56:22.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:56:22.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:22.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:56:22.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:22.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:22.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:22.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:22.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:22.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:22.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:22.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:23.853 INFO:tasks.workunit.client.0.smithi116.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-15T10:56:24.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:56:24.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:24.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:24.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:24.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:24.478 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:24.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:56:24.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:56:24.480 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:56:24.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:24.481 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:56:24.481 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:56:24.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:24.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:24.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:24.515 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:24.511+0000 7f43b060f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:24.517 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:24.515+0000 7f43b060f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:24.519 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:24.515+0000 7f43b060f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:24.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:24.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:24.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:25.062 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:25.059+0000 7f43b060f7c0 -1 Falling back to public interface 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:25.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:26.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:27.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:56:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:56:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:27.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:27.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:27.638 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:27.635+0000 7f43b060f7c0 -1 osd.0 160 log_to_monitors true 2024-06-15T10:56:28.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:28.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:28.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:56:28.452 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:56:28.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:28.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:28.793 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 164 up_thru 164 down_at 161 last_clean_interval [152,160) [v2:127.0.0.1:6802/2111552071,v1:127.0.0.1:6803/2111552071] [v2:127.0.0.1:6804/2111552071,v1:127.0.0.1:6805/2111552071] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:56:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:56:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:56:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:56:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:56:28.794 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:56:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:56:28.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:56:28.795 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:56:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:56:28.796 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:56:28.796 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:56:28.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:56:28.797 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:56:28.916 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:56:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:56:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:56:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:56:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:29.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:56:29.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636546 2024-06-15T10:56:29.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636546 2024-06-15T10:56:29.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546' 2024-06-15T10:56:29.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:29.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:56:29.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832773 2024-06-15T10:56:29.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832773 2024-06-15T10:56:29.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546 1-678604832773' 2024-06-15T10:56:29.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:29.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636546 2024-06-15T10:56:29.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:29.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:56:29.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636546 2024-06-15T10:56:29.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:29.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636546 2024-06-15T10:56:29.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636546' 2024-06-15T10:56:29.469 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 704374636546 2024-06-15T10:56:29.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 704374636546 2024-06-15T10:56:29.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:30.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:56:30.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:30.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636546 -lt 704374636546 2024-06-15T10:56:30.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:30.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832773 2024-06-15T10:56:30.989 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:30.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:56:30.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832773 2024-06-15T10:56:30.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:30.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832773 2024-06-15T10:56:30.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832773' 2024-06-15T10:56:30.995 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 678604832773 2024-06-15T10:56:30.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:31.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832773 -lt 678604832773 2024-06-15T10:56:31.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:31.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:31.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:31.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:31.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:31.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:31.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:31.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:31.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:31.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:31.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:31.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:56:31.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:31.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:31.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:56:31.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T10:56:32.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:56:32.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:32.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:32.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:32.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:32.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:32.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:32.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:32.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:56:32.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:32.318 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:32.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:32.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:56:32.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:56:32.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:56:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:56:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:56:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-06-15T10:56:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:56:32.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-06-15T10:56:33.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:56:33.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:33.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:33.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:33.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:33.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:33.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:33.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:33.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:56:33.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:33.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:33.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:56:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:56:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:56:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:56:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:56:33.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:56:33.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:56:33.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:56:33.594 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:56:33.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:56:33.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:56:33.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-06-15T10:56:33.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:56:33.881 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-06-15T10:56:34.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:56:34.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:34.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:34.285 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:34.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-06-15T10:56:34.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:34.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:34.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:34.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-06-15T10:56:34.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-06-15T10:56:34.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-06-15T10:56:34.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-06-15T10:56:34.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-06-15T10:56:34.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-06-15T10:56:34.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-06-15T10:56:34.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-06-15T10:56:34.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-06-15T10:56:35.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-06-15T10:56:35.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-06-15T10:56:35.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-06-15T10:56:35.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T10:56:35.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-06-15T10:56:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T10:56:35.879 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:35.880 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:35.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:35.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:35.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:35.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:35.881 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:36.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:36.119 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:36.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:36.120 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:36.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:36.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:36.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:37.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:38.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:38.639 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:38.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:38.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:56:38.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:38.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:38.640 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-06-15T10:56:38.641 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:56:38.641 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:38.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:38.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:38.853 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:38.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:38.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:38.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:56:38.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:56:38.855 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:56:38.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:38.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:56:38.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:56:38.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:38.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:38.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:38.891 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:38.887+0000 7ff6d3ca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:38.893 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:38.891+0000 7ff6d3ca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:38.895 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:38.891+0000 7ff6d3ca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:39.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:56:39.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:39.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:56:39.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:39.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:39.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:39.360 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:39.437 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:39.435+0000 7ff6d3ca27c0 -1 Falling back to public interface 2024-06-15T10:56:40.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:40.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:40.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:40.363 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:40.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:40.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:40.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:41.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:41.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:41.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:56:41.603 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:56:41.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:41.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:41.861 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:41.859+0000 7ff6d3ca27c0 -1 osd.1 165 log_to_monitors true 2024-06-15T10:56:41.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:56:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:56:42.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:42.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 169 up_thru 169 down_at 166 last_clean_interval [158,165) [v2:127.0.0.1:6810/3117653602,v1:127.0.0.1:6811/3117653602] [v2:127.0.0.1:6812/3117653602,v1:127.0.0.1:6813/3117653602] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:56:43.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:56:43.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:56:43.223 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:56:43.223 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:56:43.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:56:43.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:56:43.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:56:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:56:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:56:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:56:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:56:43.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:56:43.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:56:43.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:56:43.604 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:56:43.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:56:43.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:43.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:56:43.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636550 2024-06-15T10:56:43.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636550 2024-06-15T10:56:43.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636550' 2024-06-15T10:56:43.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:43.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:56:43.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473026 2024-06-15T10:56:43.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473026 2024-06-15T10:56:43.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636550 1-725849473026' 2024-06-15T10:56:43.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:43.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636550 2024-06-15T10:56:43.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:43.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:56:43.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636550 2024-06-15T10:56:43.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636550 2024-06-15T10:56:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636550' 2024-06-15T10:56:43.910 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 704374636550 2024-06-15T10:56:43.911 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:44.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636548 -lt 704374636550 2024-06-15T10:56:44.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:45.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:56:45.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:45.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636550 -lt 704374636550 2024-06-15T10:56:45.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:45.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473026 2024-06-15T10:56:45.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:45.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:56:45.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473026 2024-06-15T10:56:45.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:45.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473026 2024-06-15T10:56:45.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473026' 2024-06-15T10:56:45.389 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 725849473026 2024-06-15T10:56:45.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:45.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473026 -lt 725849473026 2024-06-15T10:56:45.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:45.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:45.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:45.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:45.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:45.915 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:46.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:46.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:46.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:46.146 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:46.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:46.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:46.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:46.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:56:46.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-06-15T10:56:46.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:46.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:46.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:46.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:47.759 INFO:tasks.workunit.client.0.smithi116.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:56:48.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:48.399 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:48.401 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:48.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:56:48.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:48.404 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:56:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:56:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:48.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:48.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:48.436 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:48.435+0000 7fc577b517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:48.439 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:48.435+0000 7fc577b517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:48.441 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:48.439+0000 7fc577b517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:48.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:48.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:48.993 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:48.991+0000 7fc577b517c0 -1 Falling back to public interface 2024-06-15T10:56:49.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:49.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:49.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:49.924 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:49.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:49.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:50.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:51.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:51.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:51.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:56:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:56:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:51.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:51.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:51.525 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:51.523+0000 7fc577b517c0 -1 osd.0 171 log_to_monitors true 2024-06-15T10:56:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:56:52.407 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:56:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:52.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:56:52.821 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 175 up_thru 175 down_at 172 last_clean_interval [164,171) [v2:127.0.0.1:6802/977809465,v1:127.0.0.1:6803/977809465] [v2:127.0.0.1:6804/977809465,v1:127.0.0.1:6805/977809465] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:56:52.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:56:52.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:56:52.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:56:52.823 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:56:52.823 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:56:52.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:56:52.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:56:52.823 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:56:52.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:56:52.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:56:52.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:56:52.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:56:52.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:56:52.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:56:53.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:56:53.220 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:56:53.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:56:53.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:53.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:56:53.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276802 2024-06-15T10:56:53.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276802 2024-06-15T10:56:53.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802' 2024-06-15T10:56:53.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:56:53.386 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:56:53.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473029 2024-06-15T10:56:53.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473029 2024-06-15T10:56:53.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802 1-725849473029' 2024-06-15T10:56:53.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:53.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276802 2024-06-15T10:56:53.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:53.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:56:53.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276802 2024-06-15T10:56:53.537 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:53.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276802 2024-06-15T10:56:53.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276802' 2024-06-15T10:56:53.538 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 751619276802 2024-06-15T10:56:53.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:53.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 751619276802 2024-06-15T10:56:53.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:56:54.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:56:54.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:56:55.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276802 -lt 751619276802 2024-06-15T10:56:55.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:56:55.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:56:55.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473029 2024-06-15T10:56:55.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:56:55.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473029 2024-06-15T10:56:55.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:56:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473029 2024-06-15T10:56:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473029' 2024-06-15T10:56:55.020 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 725849473029 2024-06-15T10:56:55.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:56:55.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473029 -lt 725849473029 2024-06-15T10:56:55.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:56:55.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:55.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:55.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:56:55.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:56:55.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:56:55.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:56:55.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:56:55.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:56:55.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:56:56.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:56:56.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:56:56.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:56:56.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:56.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:56:56.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:56:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:56:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:56:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-06-15T10:56:57.191 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:56:57.192 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:56:57.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:56:57.755 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:56:57.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:56:57.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:56:57.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:56:57.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:56:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:56:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:56:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:56:57.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:56:57.790 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:57.791+0000 7f05cebb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:57.793 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:57.795+0000 7f05cebb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:57.795 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:57.795+0000 7f05cebb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:56:57.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:58.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:58.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:56:58.330 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:56:58.331+0000 7f05cebb27c0 -1 Falling back to public interface 2024-06-15T10:56:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:56:59.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:56:59.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:56:59.257 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:56:59.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:56:59.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:56:59.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:00.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:00.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:00.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:57:00.501 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:57:00.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:00.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:00.664 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:00.667+0000 7f05cebb27c0 -1 osd.1 177 log_to_monitors true 2024-06-15T10:57:00.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:01.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:01.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:01.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:57:01.745 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:57:01.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:01.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:02.070 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [169,177) [v2:127.0.0.1:6810/1397539835,v1:127.0.0.1:6811/1397539835] [v2:127.0.0.1:6812/1397539835,v1:127.0.0.1:6813/1397539835] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:57:02.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:57:02.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:57:02.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:57:02.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:57:02.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:57:02.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:57:02.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:57:02.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:57:02.186 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:57:02.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:57:02.419 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:57:02.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:57:02.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:02.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:57:02.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276805 2024-06-15T10:57:02.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276805 2024-06-15T10:57:02.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805' 2024-06-15T10:57:02.563 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:02.564 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:57:02.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=777389080578 2024-06-15T10:57:02.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 777389080578 2024-06-15T10:57:02.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805 1-777389080578' 2024-06-15T10:57:02.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:02.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276805 2024-06-15T10:57:02.707 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:02.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:57:02.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:02.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276805 2024-06-15T10:57:02.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276805 2024-06-15T10:57:02.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276805' 2024-06-15T10:57:02.711 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 751619276805 2024-06-15T10:57:02.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:02.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276805 -lt 751619276805 2024-06-15T10:57:02.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:02.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-777389080578 2024-06-15T10:57:02.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:02.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:57:02.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-777389080578 2024-06-15T10:57:02.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:02.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=777389080578 2024-06-15T10:57:02.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2024-06-15T10:57:02.952 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 777389080578 2024-06-15T10:57:02.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 777389080578 2024-06-15T10:57:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:04.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:57:04.192 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:04.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 777389080578 2024-06-15T10:57:04.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:05.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:57:05.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:05.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 777389080578 -lt 777389080578 2024-06-15T10:57:05.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:57:05.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:05.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:05.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:57:05.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:57:05.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:57:06.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:57:06.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:57:06.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:06.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:06.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:57:06.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:57:06.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:57:06.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-15T10:57:06.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:57:06.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:57:06.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:57:06.454 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:57:06.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:57:06.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:57:06.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:57:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:57:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-06-15T10:57:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:57:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:57:06.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:57:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:57:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:06.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-06-15T10:57:07.783 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:57:07.783 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:57:08.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:57:08.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:57:08.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:57:08.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:57:08.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:57:08.320 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:08.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:08.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:08.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:57:08.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:57:08.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:57:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:57:08.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:57:08.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:57:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:57:08.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:57:08.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:57:08.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:57:08.363 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:08.363+0000 7f0b22cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:08.366 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:08.367+0000 7f0b22cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:08.367 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:08.371+0000 7f0b22cb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:08.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:57:08.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:08.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:08.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:08.922 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:08.923+0000 7f0b22cb97c0 -1 Falling back to public interface 2024-06-15T10:57:09.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:09.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:09.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:57:09.841 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:57:09.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:09.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:10.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:11.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:11.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:11.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:57:11.080 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:57:11.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:11.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:11.219 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:11.219+0000 7f0b22cb97c0 -1 osd.1 183 log_to_monitors true 2024-06-15T10:57:11.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:12.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:12.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:12.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:57:12.359 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:57:12.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:12.359 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:12.780 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 187 up_thru 187 down_at 184 last_clean_interval [181,183) [v2:127.0.0.1:6810/218755437,v1:127.0.0.1:6811/218755437] [v2:127.0.0.1:6812/218755437,v1:127.0.0.1:6813/218755437] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:57:12.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:57:12.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:57:12.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:57:12.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:57:12.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:57:12.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:57:12.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:57:12.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:57:12.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:57:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:57:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:57:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:57:13.162 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:13.163 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:57:13.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276808 2024-06-15T10:57:13.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276808 2024-06-15T10:57:13.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276808' 2024-06-15T10:57:13.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:13.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:57:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884354 2024-06-15T10:57:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884354 2024-06-15T10:57:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276808 1-803158884354' 2024-06-15T10:57:13.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:13.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276808 2024-06-15T10:57:13.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:13.468 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:57:13.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276808 2024-06-15T10:57:13.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:13.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276808 2024-06-15T10:57:13.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276808' 2024-06-15T10:57:13.471 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 751619276808 2024-06-15T10:57:13.471 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:13.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276807 -lt 751619276808 2024-06-15T10:57:13.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:14.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:57:14.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:14.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276808 -lt 751619276808 2024-06-15T10:57:14.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:14.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884354 2024-06-15T10:57:14.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:14.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:57:14.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884354 2024-06-15T10:57:14.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:14.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884354 2024-06-15T10:57:14.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884354' 2024-06-15T10:57:14.964 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 803158884354 2024-06-15T10:57:14.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:15.178 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884354 -lt 803158884354 2024-06-15T10:57:15.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:57:15.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:15.180 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:15.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:57:15.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:57:15.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:57:15.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:57:15.418 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:57:15.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:57:15.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:57:15.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:57:15.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:57:15.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:57:15.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:15.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-06-15T10:57:15.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-06-15T10:57:15.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-06-15T10:57:15.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-06-15T10:57:15.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-06-15T10:57:15.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-15T10:57:15.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-15T10:57:15.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-15T10:57:15.891 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:57:16.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-06-15T10:57:16.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-06-15T10:57:16.257 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: {} 2024-06-15T10:57:16.257 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-15T10:57:16.273 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: {} 2024-06-15T10:57:16.274 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:57:16.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:57:16.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-15T10:57:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-15T10:57:16.290 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-15T10:57:16.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:57:16.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:57:16.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:57:16.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:57:16.541 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:57:16.541 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:57:16.541 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:57:16.541 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:57:16.542 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:16.542 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:16.542 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:16.542 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:57:16.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:57:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:57:16.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:57:16.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:57:16.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-15T10:57:16.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:57:16.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:57:16.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-15T10:57:16.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-15T10:57:16.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:57:16.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:57:16.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-15T10:57:16.791 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:57:17.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:57:17.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-15T10:57:17.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:57:17.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:57:17.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:57:17.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-15T10:57:17.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:57:17.024 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:57:17.025 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:17.025 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:17.025 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:17.025 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:57:17.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:57:17.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:57:17.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:57:17.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:57:17.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-15T10:57:17.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:57:17.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:57:17.152 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:17.153 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:17.153 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:17.153 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:57:17.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:57:17.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-15T10:57:17.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-15T10:57:17.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-15T10:57:17.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:57:17.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-15T10:57:17.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:57:17.269 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-15T10:57:17.269 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-15T10:57:17.269 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:57:17.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:57:17.498 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:57:17.498 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:57:17.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:57:17.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:57:17.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:57:17.502 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-15T10:57:17.502 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:57:17.502 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:57:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:17.503 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:57:17.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:57:17.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:57:17.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:57:17.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:57:17.620 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-15T10:57:17.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:57:17.621 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:57:17.621 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:17.622 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:17.622 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:17.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:57:17.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:57:17.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:57:17.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:57:17.743 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-15T10:57:17.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:57:17.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-15T10:57:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-15T10:57:17.744 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:57:17.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:57:17.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-15T10:57:17.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:57:17.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:57:17.966 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:57:17.966 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:57:17.966 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:57:17.967 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:17.967 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:17.967 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:17.967 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:57:17.967 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:57:18.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:57:18.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:57:18.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:57:18.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-15T10:57:18.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:57:18.073 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:57:18.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-15T10:57:18.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-06-15T10:57:18.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-15T10:57:18.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-15T10:57:18.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:18.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:18.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:18.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:18.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:18.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-15T10:57:18.614 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-15T10:57:18.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:18.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:57:18.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:18.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:57:18.630 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:57:18.631 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:57:18.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:57:18.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:18.631 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:18.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:18.632 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:18.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:18.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:57:19.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:57:19.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:57:19.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:57:19.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:19.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:19.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:19.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:20.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:20.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:57:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:57:21.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:57:21.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:57:21.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:21.099 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:21.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:21.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:21.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:21.345 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:57:22.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:57:22.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:57:22.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:57:22.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:22.347 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:22.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:22.348 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:22.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:22.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:57:23.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:57:23.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:57:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:57:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:57:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:57:23.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:57:23.588 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:57:23.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:57:19.199587+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-15T10:57:23.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-15T10:57:23.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-15T10:57:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-06-15T10:57:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-06-15T10:57:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-06-15T10:57:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.923 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:23.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:23.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:57:24.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-06-15T10:57:24.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-15T10:57:24.122 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:57:24.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-15T10:57:24.152 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:57:24.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-15T10:57:24.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-15T10:57:24.211 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=187 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:57:24.240 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-15T10:57:24.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:57:24.287 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-15T10:57:24.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:57:24.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:57:24.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:57:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-06-15T10:57:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-15T10:57:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-06-15T10:57:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-06-15T10:57:24.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:57:24.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:57:24.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:57:24.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:57:24.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:57:24.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:57:24.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:57:24.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:57:24.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:57:24.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:57:24.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:57:24.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-06-15T10:57:25.886 INFO:tasks.workunit.client.0.smithi116.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-15T10:57:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:57:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:57:26.423 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:57:26.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:57:26.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:57:26.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:57:26.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:57:26.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:57:26.426 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:57:26.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:57:26.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:57:26.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:57:26.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:57:26.428 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:57:26.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:57:26.429 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:57:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:57:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:57:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:57:26.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:57:26.461 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:26.462+0000 7f09417907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:26.464 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:26.466+0000 7f09417907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:26.465 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:26.466+0000 7f09417907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:26.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:57:26.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:57:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:26.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:26.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:27.006 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:27.006+0000 7f09417907c0 -1 Falling back to public interface 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:27.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:28.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:57:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:57:29.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:29.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:29.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:29.589 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:29.590+0000 7f09417907c0 -1 osd.0 189 log_to_monitors true 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:30.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 193 up_thru 193 down_at 190 last_clean_interval [175,189) [v2:127.0.0.1:6802/1270678121,v1:127.0.0.1:6803/1270678121] [v2:127.0.0.1:6804/1270678121,v1:127.0.0.1:6805/1270678121] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:57:30.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:57:30.850 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:57:30.850 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:57:30.850 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:57:30.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:57:30.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:57:30.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:57:30.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:57:30.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:57:30.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:57:30.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:57:30.980 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:57:30.980 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:57:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:57:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:57:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:57:31.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:31.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:57:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688130 2024-06-15T10:57:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688130 2024-06-15T10:57:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130' 2024-06-15T10:57:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:31.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:57:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884359 2024-06-15T10:57:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884359 2024-06-15T10:57:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130 1-803158884359' 2024-06-15T10:57:31.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:31.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688130 2024-06-15T10:57:31.665 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:31.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:57:31.666 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688130 2024-06-15T10:57:31.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:31.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688130 2024-06-15T10:57:31.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688130' 2024-06-15T10:57:31.668 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 828928688130 2024-06-15T10:57:31.668 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:31.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 828928688130 2024-06-15T10:57:31.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:32.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:57:32.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:33.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688130 -lt 828928688130 2024-06-15T10:57:33.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:33.138 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:33.139 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884359 2024-06-15T10:57:33.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:57:33.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884359 2024-06-15T10:57:33.141 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:33.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884359 2024-06-15T10:57:33.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884359' 2024-06-15T10:57:33.144 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 803158884359 2024-06-15T10:57:33.144 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:33.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884359 -lt 803158884359 2024-06-15T10:57:33.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:57:33.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:33.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:33.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:57:33.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:57:33.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:57:33.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:57:33.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:57:33.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:33.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:57:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:57:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:57:34.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-06-15T10:57:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-06-15T10:57:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-06-15T10:57:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-15T10:57:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:57:34.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:57:34.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:57:34.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:57:34.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:57:34.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:57:35.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:57:35.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:35.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-06-15T10:57:36.263 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:57:36.263 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:57:36.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:36.801 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:36.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:36.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:36.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:57:36.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:57:36.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:57:36.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:57:36.804 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:57:36.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:57:36.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:57:36.804 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:57:36.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:57:36.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:57:36.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:57:36.836 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:36.838+0000 7fec8d9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:36.845 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:36.846+0000 7fec8d9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:36.847 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:36.846+0000 7fec8d9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:37.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:57:37.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:57:37.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:57:37.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:57:37.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:57:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:57:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:57:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:37.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:37.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:37.382 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:37.382+0000 7fec8d9357c0 -1 Falling back to public interface 2024-06-15T10:57:38.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:38.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:38.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:57:38.292 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:57:38.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:38.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:38.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:39.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:39.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:39.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:57:39.535 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:57:39.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:39.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:39.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:39.804 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:39.778+0000 7fec8d9357c0 -1 osd.1 195 log_to_monitors true 2024-06-15T10:57:40.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:40.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:40.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:57:40.776 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:57:40.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:40.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:41.325 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 199 up_thru 199 down_at 196 last_clean_interval [187,195) [v2:127.0.0.1:6810/1812407926,v1:127.0.0.1:6811/1812407926] [v2:127.0.0.1:6812/1812407926,v1:127.0.0.1:6813/1812407926] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:57:41.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:57:41.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:57:41.326 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:57:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:57:41.327 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:57:41.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:57:41.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:57:41.328 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:57:41.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:57:41.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:57:41.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:57:41.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:57:41.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:57:41.453 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:57:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:57:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:57:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:57:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:41.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:57:41.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688133 2024-06-15T10:57:41.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688133 2024-06-15T10:57:41.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133' 2024-06-15T10:57:41.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:41.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:57:42.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491906 2024-06-15T10:57:42.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491906 2024-06-15T10:57:42.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133 1-854698491906' 2024-06-15T10:57:42.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:42.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688133 2024-06-15T10:57:42.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:42.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:57:42.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688133 2024-06-15T10:57:42.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:42.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688133 2024-06-15T10:57:42.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688133' 2024-06-15T10:57:42.080 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 828928688133 2024-06-15T10:57:42.080 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688132 -lt 828928688133 2024-06-15T10:57:42.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:43.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:57:43.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:43.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688133 -lt 828928688133 2024-06-15T10:57:43.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:43.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491906 2024-06-15T10:57:43.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:43.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:57:43.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491906 2024-06-15T10:57:43.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:43.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491906 2024-06-15T10:57:43.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491906' 2024-06-15T10:57:43.560 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 854698491906 2024-06-15T10:57:43.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:43.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491906 -lt 854698491906 2024-06-15T10:57:43.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:57:43.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:43.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:44.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:57:44.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:57:44.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:57:44.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:57:44.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:57:44.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:57:44.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:57:44.110 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:57:44.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:57:44.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:57:44.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:44.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:57:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:57:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:57:44.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-06-15T10:57:44.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:44.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:57:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:57:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:57:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:57:44.750 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:57:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:57:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:57:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:57:44.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:57:44.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:57:44.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:57:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:57:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:57:44.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:57:46.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:57:46.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:46.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:57:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:57:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T10:57:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:57:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:57:46.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:47.111 INFO:tasks.workunit.client.0.smithi116.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-06-15T10:57:47.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:57:47.647 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:57:47.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:57:47.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:57:47.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:57:47.649 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:57:47.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T10:57:47.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:57:47.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T10:57:47.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T10:57:47.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:57:47.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:57:47.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:57:47.686 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:47.686+0000 7fa8200467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:47.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:47.690+0000 7fa8200467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:47.690 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:47.690+0000 7fa8200467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:47.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:57:47.893 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:57:47.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:47.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:48.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:48.250 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:48.250+0000 7fa8200467c0 -1 Falling back to public interface 2024-06-15T10:57:49.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:49.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:49.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:57:49.130 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:57:49.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:49.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:49.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:50.247 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:50.246+0000 7fa8200467c0 -1 osd.0 201 log_to_monitors true 2024-06-15T10:57:50.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:50.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:50.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:57:50.373 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:57:50.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:50.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:50.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:51.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:57:51.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:51.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:57:51.710 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:57:51.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:51.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:57:52.073 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 205 up_thru 205 down_at 202 last_clean_interval [193,201) [v2:127.0.0.1:6802/1105790913,v1:127.0.0.1:6803/1105790913] [v2:127.0.0.1:6804/1105790913,v1:127.0.0.1:6805/1105790913] exists,up 9b098f5b-3fdb-47fa-97cc-efb99630c9df 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:57:52.074 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:57:52.076 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:57:52.076 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:57:52.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:57:52.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:57:52.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:57:52.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:57:52.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:57:52.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:57:52.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:57:52.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:57:52.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:57:52.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:57:52.463 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:57:52.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:57:52.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:52.464 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:57:52.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295682 2024-06-15T10:57:52.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295682 2024-06-15T10:57:52.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682' 2024-06-15T10:57:52.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:57:52.612 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:57:52.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491909 2024-06-15T10:57:52.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491909 2024-06-15T10:57:52.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682 1-854698491909' 2024-06-15T10:57:52.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:52.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295682 2024-06-15T10:57:52.765 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:52.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:57:52.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295682 2024-06-15T10:57:52.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:52.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295682 2024-06-15T10:57:52.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295682' 2024-06-15T10:57:52.768 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 880468295682 2024-06-15T10:57:52.768 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:57:52.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295682 -lt 880468295682 2024-06-15T10:57:52.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:57:52.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:57:52.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491909 2024-06-15T10:57:52.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:57:52.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491909 2024-06-15T10:57:52.999 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:57:53.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491909 2024-06-15T10:57:53.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491909' 2024-06-15T10:57:53.000 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 854698491909 2024-06-15T10:57:53.001 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:53.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491908 -lt 854698491909 2024-06-15T10:57:53.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:54.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:57:54.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:54.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491908 -lt 854698491909 2024-06-15T10:57:54.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:57:55.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:57:55.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:57:55.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491909 -lt 854698491909 2024-06-15T10:57:55.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:57:55.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:55.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:55.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:57:55.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:57:55.951 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:57:55.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:57:55.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:57:55.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:57:55.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:57:55.952 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:57:56.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:57:56.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:57:56.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:57:56.231 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:57:56.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:57:56.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:57:56.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:57:56.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:57:56.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:57:56.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:57:56.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:57:56.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:57:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:57.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-06-15T10:57:58.465 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-06-15T10:57:58.466 INFO:tasks.workunit.client.0.smithi116.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-06-15T10:57:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:57:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:57:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:57:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:57:59.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:57:59.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:57:59.005 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:57:59.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:57:59.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:57:59.006 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:57:59.006 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:59.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:57:59.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:57:59.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:57:59.008 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:57:59.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:57:59.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:57:59.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:57:59.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:57:59.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:57:59.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:57:59.042 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:59.042+0000 7ff6261ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:59.044 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:59.046+0000 7ff6261ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:59.046 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:59.046+0000 7ff6261ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:57:59.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:57:59.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:57:59.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:57:59.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:57:59.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:57:59.597 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:57:59.598+0000 7ff6261ad7c0 -1 Falling back to public interface 2024-06-15T10:58:00.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:58:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:58:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:00.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:58:00.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:01.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:58:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:58:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:01.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:58:01.985 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:01.986+0000 7ff6261ad7c0 -1 osd.1 207 log_to_monitors true 2024-06-15T10:58:02.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:03.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:03.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:03.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:58:03.027 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:58:03.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:03.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:58:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 211 up_thru 211 down_at 208 last_clean_interval [199,207) [v2:127.0.0.1:6810/1519105237,v1:127.0.0.1:6811/1519105237] [v2:127.0.0.1:6812/1519105237,v1:127.0.0.1:6813/1519105237] exists,up e86e26b4-2c61-4754-b772-63f282ebe161 2024-06-15T10:58:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:58:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:58:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:58:03.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:58:03.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:58:03.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:58:03.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:58:03.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:58:03.477 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:58:03.477 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:58:03.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:58:03.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:58:03.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:58:03.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:58:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:58:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:58:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:58:03.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:58:03.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:58:03.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:58:03.869 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:58:03.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:58:03.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:58:03.869 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:58:04.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295685 2024-06-15T10:58:04.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295685 2024-06-15T10:58:04.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685' 2024-06-15T10:58:04.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:58:04.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:58:04.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099458 2024-06-15T10:58:04.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099458 2024-06-15T10:58:04.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685 1-906238099458' 2024-06-15T10:58:04.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:58:04.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295685 2024-06-15T10:58:04.203 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:58:04.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:58:04.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295685 2024-06-15T10:58:04.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:58:04.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295685 2024-06-15T10:58:04.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295685' 2024-06-15T10:58:04.206 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 880468295685 2024-06-15T10:58:04.206 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:58:04.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295684 -lt 880468295685 2024-06-15T10:58:04.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:58:05.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:58:05.438 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:58:05.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295685 -lt 880468295685 2024-06-15T10:58:05.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:58:05.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099458 2024-06-15T10:58:05.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:58:05.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:58:05.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099458 2024-06-15T10:58:05.683 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:58:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099458 2024-06-15T10:58:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099458' 2024-06-15T10:58:05.684 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 906238099458 2024-06-15T10:58:05.684 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:58:05.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099458 -lt 906238099458 2024-06-15T10:58:05.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:58:05.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:58:05.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:58:06.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:58:06.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:58:06.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:58:06.448 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:58:06.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:58:06.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:58:06.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:58:06.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:58:06.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:58:06.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:58:06.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-06-15T10:58:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-06-15T10:58:06.711 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-06-15T10:58:06.712 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:58:06.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:58:06.944 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:58:06.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:58:06.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:58:06.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:58:06.947 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:58:06.947 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:58:06.947 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:58:06.948 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:06.948 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:06.948 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:06.948 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:58:06.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:58:07.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:58:07.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:58:07.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:58:07.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-15T10:58:07.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:58:07.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:58:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:07.071 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:07.071 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:58:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:58:07.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:58:07.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-15T10:58:07.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:58:07.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-06-15T10:58:07.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:58:07.198 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-06-15T10:58:07.198 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-06-15T10:58:07.198 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:58:07.444 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:58:07.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:58:07.446 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:58:07.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:58:07.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:58:07.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-15T10:58:07.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:58:07.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:58:07.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:58:07.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:07.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:07.563 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:07.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:58:07.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:58:07.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-06-15T10:58:07.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-06-15T10:58:07.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-06-15T10:58:07.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:58:07.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-15T10:58:07.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:58:07.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-15T10:58:07.675 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:58:07.675 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:58:07.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-06-15T10:58:07.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:58:07.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:58:07.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-06-15T10:58:07.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:58:07.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:58:07.908 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:58:07.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:58:08.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:58:08.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:58:08.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:08.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.0.asok 2024-06-15T10:58:08.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:58:08.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-06-15T10:58:08.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-06-15T10:58:08.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-06-15T10:58:08.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-06-15T10:58:08.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-06-15T10:58:08.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-06-15T10:58:08.148 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-06-15T10:58:08.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-06-15T10:58:08.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-06-15T10:58:08.149 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-06-15T10:58:08.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-06-15T10:58:08.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-06-15T10:58:08.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-06-15T10:58:08.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-06-15T10:58:08.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-06-15T10:58:08.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-06-15T10:58:08.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:08.387 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:58:08.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.62093/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-06-15T10:58:08.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-06-15T10:58:08.500 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-06-15T10:58:08.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-06-15T10:58:08.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-06-15T10:58:08.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-06-15T10:58:08.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-06-15T10:58:08.502 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:08.503 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:08.503 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:08.503 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-osd.1.asok 2024-06-15T10:58:08.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-06-15T10:58:08.503 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.62093/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-06-15T10:58:08.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-06-15T10:58:08.773 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0: {} 2024-06-15T10:58:08.773 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0: osd_max_object_size = '' 2024-06-15T10:58:08.791 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1: {} 2024-06-15T10:58:08.791 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1: osd_max_object_size = '' 2024-06-15T10:58:08.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-06-15T10:58:08.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-06-15T10:58:08.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:58:08.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:08.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:58:08.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:08.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:58:09.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:09.041 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-06-15T10:58:09.252 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-06-15T10:53:22.460704+0000 last_deep_scrub_stamp 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:58:09.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:09.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:58:09.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:09.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:58:09.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:09.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:58:09.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:09.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:10.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:10.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:10.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:58:10.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:10.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:58:10.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:10.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:58:10.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:10.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:11.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:11.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:11.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:58:11.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:11.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:58:11.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:11.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:58:11.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:53:22.460704+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:11.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:12.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:12.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:13.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-06-15T10:58:13.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:13.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T10:58:13.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-06-15T10:58:13.000 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:13.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:10.048761+0000 '>' 2024-06-15T10:53:22.460704+0000 2024-06-15T10:58:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:58:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-06-15T10:58:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-06-15T10:58:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-06-15T10:58:13.247 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-06-15T10:58:13.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-06-15T10:58:13.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-06-15T10:58:13.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.367 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.377 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.638 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-06-15T10:58:13.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-06-15T10:58:13.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-06-15T10:58:13.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-06-15T10:58:13.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-06-15T10:58:13.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T10:58:13.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-06-15T10:58:13.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-06-15T10:58:13.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-06-15T10:58:13.883 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=211 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:58:13.884 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-15T10:58:13.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T10:58:13.938 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-06-15T10:58:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T10:58:13.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:58:13.991 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T10:58:13.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-06-15T10:58:13.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-06-15T10:58:13.992 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-06-15T10:58:13.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-06-15T10:58:13.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-06-15T10:58:13.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:13.993 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:13.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:13.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:14.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-06-15T10:58:14.411 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0 on osd.1 to repair 2024-06-15T10:58:14.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:14.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:58:14.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:14.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:14.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:14.427 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:10.048761+0000 '>' 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:14.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:15.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:15.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:15.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:58:15.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:15.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:15.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:15.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:15.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:10.048761+0000 '>' 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:15.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:16.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:16.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:16.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:58:16.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:16.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:16.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:16.889 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:17.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:10.048761+0000 '>' 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:17.128 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:18.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:18.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:18.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:58:18.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:18.131 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:18.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:18.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:10.048761+0000 '>' 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T10:58:19.370 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T10:58:19.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T10:58:19.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T10:58:19.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T10:58:19.372 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T10:58:19.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T10:58:19.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:58:15.692556+0000 '>' 2024-06-15T10:58:10.048761+0000 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:58:19.615 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:58:19.616 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:58:19.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:58:19.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:58:19.962 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:58:19.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:58:19.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:58:19.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:58:21.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295690 2024-06-15T10:58:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295690 2024-06-15T10:58:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295690' 2024-06-15T10:58:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:58:21.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:58:21.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099462 2024-06-15T10:58:21.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099462 2024-06-15T10:58:21.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295690 1-906238099462' 2024-06-15T10:58:21.209 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:58:21.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295690 2024-06-15T10:58:21.210 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:58:21.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:58:21.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295690 2024-06-15T10:58:21.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:58:21.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295690 2024-06-15T10:58:21.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295690' 2024-06-15T10:58:21.214 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 880468295690 2024-06-15T10:58:21.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:58:21.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295689 -lt 880468295690 2024-06-15T10:58:21.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:58:22.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:58:22.449 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:58:22.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295689 -lt 880468295690 2024-06-15T10:58:22.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:58:23.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T10:58:23.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:58:23.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295690 -lt 880468295690 2024-06-15T10:58:23.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:58:23.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099462 2024-06-15T10:58:23.938 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:58:23.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:58:23.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099462 2024-06-15T10:58:23.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:58:23.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099462 2024-06-15T10:58:23.943 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 906238099462 2024-06-15T10:58:23.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099462' 2024-06-15T10:58:23.943 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:58:24.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099463 -lt 906238099462 2024-06-15T10:58:24.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:58:24.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:58:24.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:58:24.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:58:24.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:58:24.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:58:24.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:58:24.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:58:24.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:58:24.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:58:24.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:58:24.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:58:24.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:58:24.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:58:24.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:58:24.983 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:58:24.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:58:24.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:58:24.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-15T10:58:25.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-15T10:58:25.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-15T10:58:25.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-06-15T10:58:25.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-15T10:58:25.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-06-15T10:58:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-06-15T10:58:25.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-15T10:58:25.369 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'csr_pool' removed 2024-06-15T10:58:25.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T10:58:25.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:58:25.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:58:25.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:58:25.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:58:25.389 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:58:25.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:58:25.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:58:25.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:58:25.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:58:25.703 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:58:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:58:25.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:58:25.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:58:25.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:58:25.708 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:58:25.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:58:25.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:58:25.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:58:25.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:58:25.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:58:25.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:58:25.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:58:25.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:58:25.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:25.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:25.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:58:25.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:58:25.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:58:25.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:58:25.769 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T10:58:25.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T10:58:25.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T10:58:25.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T10:58:25.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T10:58:25.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T10:58:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:58:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T10:58:25.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T10:58:25.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T10:58:25.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T10:58:25.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T10:58:25.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T10:58:25.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T10:58:25.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:25.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:25.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T10:58:25.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T10:58:25.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T10:58:25.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T10:58:25.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T10:58:25.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:25.782 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:25.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T10:58:25.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T10:58:25.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T10:58:25.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T10:58:25.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-15T10:58:26.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T10:58:26.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:58:26.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:58:26.172 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:58:26.172 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:26.172 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:26.173 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:58:26.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-15T10:58:26.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T10:58:26.392 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T10:58:26.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:58:26.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:58:26.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T10:58:26.393 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T10:58:26.394 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:58:26.394 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:58:26.394 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:58:26.395 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T10:58:26.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T10:58:26.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T10:58:26.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T10:58:26.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T10:58:26.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T10:58:26.512 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:26.513 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T10:58:26.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T10:58:26.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T10:58:26.623 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T10:58:27.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T10:58:27.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:58:27.482 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:58:27.483 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:58:27.483 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:27.483 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:27.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:58:27.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:58:27.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:58:27.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:58:27.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:58:27.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bacb953a-786c-41cf-80f0-a760c0793fde 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bacb953a-786c-41cf-80f0-a760c0793fde' 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 bacb953a-786c-41cf-80f0-a760c0793fde 2024-06-15T10:58:27.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:58:27.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTc21mEzGSIBAAh1yR1d3HHMvkpzJ/UKMNvA== 2024-06-15T10:58:27.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTc21mEzGSIBAAh1yR1d3HHMvkpzJ/UKMNvA=="}' 2024-06-15T10:58:27.544 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bacb953a-786c-41cf-80f0-a760c0793fde -i td/osd-scrub-repair/0/new.json 2024-06-15T10:58:27.785 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:58:27.801 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T10:58:27.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTc21mEzGSIBAAh1yR1d3HHMvkpzJ/UKMNvA== --osd-uuid bacb953a-786c-41cf-80f0-a760c0793fde 2024-06-15T10:58:27.834 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.833+0000 7f1bdc2207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:27.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.837+0000 7f1bdc2207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:27.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.837+0000 7f1bdc2207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:27.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.837+0000 7f1bdc2207c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:58:27.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.837+0000 7f1bdc2207c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:58:27.838 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.837+0000 7f1bdc2207c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T10:58:27.927 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:27.925+0000 7f1bdc2207c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T10:58:33.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T10:58:33.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:58:33.584 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T10:58:33.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T10:58:33.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:58:33.852 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T10:58:33.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T10:58:33.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:58:33.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:58:33.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:58:33.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:58:33.884 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:33.885+0000 7f78085e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:33.886 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:33.885+0000 7f78085e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:33.887 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:33.889+0000 7f78085e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:58:34.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:34.083 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:58:34.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:58:34.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:34.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:34.426 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:34.425+0000 7f78085e97c0 -1 Falling back to public interface 2024-06-15T10:58:35.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:35.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:35.334 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:58:35.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:58:35.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:35.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:36.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:36.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:36.579 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:58:36.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:58:36.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:36.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:36.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:36.989 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:58:36.989+0000 7f78085e97c0 -1 osd.0 0 log_to_monitors true 2024-06-15T10:58:37.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:37.816 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:58:37.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:37.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:58:37.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:37.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:38.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:39.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:39.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:39.155 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:58:39.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:58:39.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:39.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:39.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:40.397 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:40.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:40.398 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:58:40.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:58:40.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:40.398 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:40.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:41.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:41.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:41.661 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:58:41.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:58:41.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:41.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:41.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:42.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:42.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:42.986 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:58:42.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:58:42.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:42.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:43.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:44.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:44.231 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:58:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:58:44.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:44.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:44.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:45.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:45.472 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:58:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:58:45.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:45.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:45.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:46.812 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:58:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:58:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:46.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:47.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:48.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:48.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:48.056 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:58:48.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:58:48.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:48.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:48.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:49.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:49.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:49.298 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:58:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:58:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:49.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:49.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:50.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:50.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:50.913 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:58:50.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:58:50.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:50.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:51.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:52.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:52.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:52.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:58:52.343 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:58:52.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:52.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:52.936 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:53.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:53.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:53.938 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:58:53.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:58:53.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:53.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:54.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:55.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:55.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:55.769 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:58:55.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:58:55.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:55.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:56.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:57.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:57.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:57.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:58:57.408 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:58:57.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:57.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:57.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:58:58.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:58:58.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:58:58.933 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:58:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:58:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:58:58.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:58:59.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:00.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:00.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:00.214 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:59:00.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:59:00.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:00.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T10:59:00.452 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/155029846,v1:127.0.0.1:6803/155029846] [v2:127.0.0.1:6804/155029846,v1:127.0.0.1:6805/155029846] exists,up bacb953a-786c-41cf-80f0-a760c0793fde 2024-06-15T10:59:00.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:59:00.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:59:00.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:59:00.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:59:00.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:59:00.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T10:59:00.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=195879a4-0386-46c8-a916-79b7d954c749 2024-06-15T10:59:00.460 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 195879a4-0386-46c8-a916-79b7d954c749 2024-06-15T10:59:00.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 195879a4-0386-46c8-a916-79b7d954c749' 2024-06-15T10:59:00.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T10:59:00.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0c21moG7eHBAAhJ+UlMs5d/z4W/EyOw/K3A== 2024-06-15T10:59:00.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0c21moG7eHBAAhJ+UlMs5d/z4W/EyOw/K3A=="}' 2024-06-15T10:59:00.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 195879a4-0386-46c8-a916-79b7d954c749 -i td/osd-scrub-repair/1/new.json 2024-06-15T10:59:00.768 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:59:00.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T10:59:00.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0c21moG7eHBAAhJ+UlMs5d/z4W/EyOw/K3A== --osd-uuid 195879a4-0386-46c8-a916-79b7d954c749 2024-06-15T10:59:00.824 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.825+0000 7f86254a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:00.835 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.837+0000 7f86254a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:00.836 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.837+0000 7f86254a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:00.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.837+0000 7f86254a97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:59:00.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.837+0000 7f86254a97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:59:00.837 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.837+0000 7f86254a97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T10:59:00.904 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:00.905+0000 7f86254a97c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T10:59:06.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T10:59:06.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T10:59:06.694 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T10:59:06.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T10:59:06.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T10:59:06.968 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T10:59:06.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T10:59:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:59:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T10:59:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T10:59:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T10:59:07.001 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:07.001+0000 7f36dbd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:07.039 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:07.041+0000 7f36dbd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:07.041 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:07.041+0000 7f36dbd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:07.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:59:07.232 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T10:59:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:07.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:07.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:07.590 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:07.589+0000 7f36dbd3a7c0 -1 Falling back to public interface 2024-06-15T10:59:08.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:08.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:08.483 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T10:59:08.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:59:08.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:08.484 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:08.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:09.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:09.720 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:09.720 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T10:59:09.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:59:09.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:09.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:09.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:10.233 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:10.233+0000 7f36dbd3a7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T10:59:10.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:10.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:10.959 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T10:59:10.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:59:10.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:10.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:11.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:12.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:12.293 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T10:59:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T10:59:12.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:12.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:12.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:13.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:13.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:13.536 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T10:59:13.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T10:59:13.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:13.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:13.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:14.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:14.813 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T10:59:14.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T10:59:14.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:14.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:15.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:16.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:16.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:16.213 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T10:59:16.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T10:59:16.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:16.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:16.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:17.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:17.455 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T10:59:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T10:59:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:17.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:17.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:18.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:18.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:18.686 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T10:59:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T10:59:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:18.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:19.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:20.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:20.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:20.003 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T10:59:20.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T10:59:20.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:20.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:20.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:21.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:21.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:21.283 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T10:59:21.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T10:59:21.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:21.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:21.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:22.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:22.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:22.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T10:59:22.527 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T10:59:22.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:22.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:22.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:23.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:23.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:23.914 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T10:59:23.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T10:59:23.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:23.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:24.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:25.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:25.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:25.334 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T10:59:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T10:59:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:25.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:25.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:26.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:26.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:26.966 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T10:59:26.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T10:59:26.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:26.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:27.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:28.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:28.643 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:28.643 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T10:59:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T10:59:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:28.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:29.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:30.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:30.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:30.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T10:59:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T10:59:30.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:30.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:30.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:31.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:31.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:31.617 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T10:59:31.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T10:59:31.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:31.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:32.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:33.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:33.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:33.368 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T10:59:33.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T10:59:33.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:33.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:33.610 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3864009768,v1:127.0.0.1:6811/3864009768] [v2:127.0.0.1:6812/3864009768,v1:127.0.0.1:6813/3864009768] exists,up 195879a4-0386-46c8-a916-79b7d954c749 2024-06-15T10:59:33.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:59:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:59:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:59:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-06-15T10:59:33.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T10:59:33.824 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T10:59:33.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T10:59:33.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T10:59:34.224 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T10:59:34.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:59:35.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T10:59:36.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-15T10:59:36.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:59:36.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:59:36.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:59:36.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:59:36.787 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:59:36.787 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:59:36.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:59:36.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:59:36.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:59:36.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:59:36.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:59:36.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:59:36.928 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:59:36.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:59:36.929 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:59:37.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:59:37.168 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:59:37.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:59:37.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:37.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:59:37.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-06-15T10:59:37.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-06-15T10:59:37.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-06-15T10:59:37.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:37.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:59:37.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-06-15T10:59:37.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-06-15T10:59:37.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672965' 2024-06-15T10:59:37.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:37.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-06-15T10:59:37.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:37.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:59:37.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-06-15T10:59:37.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:37.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-06-15T10:59:37.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-15T10:59:37.478 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836494 2024-06-15T10:59:37.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:37.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-06-15T10:59:37.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:59:38.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:59:38.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:38.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-15T10:59:38.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:38.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-06-15T10:59:38.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:38.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:59:38.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-06-15T10:59:38.960 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:38.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-06-15T10:59:38.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-06-15T10:59:38.962 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672965 2024-06-15T10:59:38.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:59:39.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-06-15T10:59:39.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:59:39.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:39.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:39.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T10:59:39.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:59:39.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:59:39.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:59:39.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:59:39.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:59:39.479 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:59:39.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:59:39.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T10:59:39.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:59:39.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:39.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T10:59:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:59:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:59:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-06-15T10:59:39.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-06-15T10:59:40.316 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'foo' created 2024-06-15T10:59:40.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:59:41.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-06-15T10:59:41.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-06-15T10:59:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'csr_pool' created 2024-06-15T10:59:41.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T10:59:42.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-06-15T10:59:42.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:59:42.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:59:42.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:59:42.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:59:42.700 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:59:42.700 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:59:42.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:59:42.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:59:42.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:59:42.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:59:42.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:59:42.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:59:42.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:59:42.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:59:42.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:59:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:59:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:59:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:59:43.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:43.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:59:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-06-15T10:59:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-06-15T10:59:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-06-15T10:59:43.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:43.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:59:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-06-15T10:59:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-06-15T10:59:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672967' 2024-06-15T10:59:43.376 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:43.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-06-15T10:59:43.377 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:43.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:59:43.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-06-15T10:59:43.380 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-06-15T10:59:43.381 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836496 2024-06-15T10:59:43.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-06-15T10:59:43.381 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:43.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-06-15T10:59:43.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:59:44.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:59:44.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-06-15T10:59:44.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:44.852 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-06-15T10:59:44.853 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:44.855 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:59:44.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-06-15T10:59:44.856 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:44.857 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-06-15T10:59:44.858 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672967 2024-06-15T10:59:44.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-06-15T10:59:44.858 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:59:45.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-06-15T10:59:45.102 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:59:45.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:45.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:45.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:59:45.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:59:45.359 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:59:45.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:59:45.590 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:59:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:45.591 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:45.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:59:45.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:59:45.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:59:45.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:59:45.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:59:46.172 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:59:46.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:59:46.691 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:59:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:59:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:59:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:59:46.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-06-15T10:59:47.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-06-15T10:59:47.205 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T10:59:47.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T10:59:47.543 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T10:59:47.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T10:59:48.053 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T10:59:48.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T10:59:48.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T10:59:48.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-06-15T10:59:48.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-06-15T10:59:48.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-06-15T10:59:48.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-06-15T10:59:48.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-06-15T10:59:48.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-06-15T10:59:48.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-06-15T10:59:48.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T10:59:48.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-06-15T10:59:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-06-15T10:59:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-06-15T10:59:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-06-15T10:59:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T10:59:48.801 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-06-15T10:59:49.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-06-15T10:59:49.050 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-06-15T10:59:49.277 INFO:tasks.workunit.client.0.smithi116.stdout:created pool csr_pool snap snap1 2024-06-15T10:59:49.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-06-15T10:59:49.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-06-15T10:59:49.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-15T10:59:49.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-06-15T10:59:49.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-06-15T10:59:49.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-06-15T10:59:49.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-15T10:59:49.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:59:49.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:59:49.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-06-15T10:59:49.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:59:49.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T10:59:49.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:59:49.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:59:49.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:59:49.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:59:49.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T10:59:49.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:59:49.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-06-15T10:59:51.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T10:59:51.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T10:59:51.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T10:59:51.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T10:59:51.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T10:59:51.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T10:59:51.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T10:59:51.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T10:59:51.831 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T10:59:51.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T10:59:51.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T10:59:51.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T10:59:51.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T10:59:51.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T10:59:51.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T10:59:51.866 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:51.864+0000 7f796ef967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:51.996 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:51.996+0000 7f796ef967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:51.998 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:51.996+0000 7f796ef967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T10:59:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T10:59:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T10:59:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T10:59:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T10:59:52.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T10:59:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T10:59:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T10:59:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:52.121 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:52.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:52.542 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:52.540+0000 7f796ef967c0 -1 Falling back to public interface 2024-06-15T10:59:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T10:59:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T10:59:53.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:53.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:53.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T10:59:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T10:59:54.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:54.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:54.896 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T10:59:55.259 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T10:59:55.256+0000 7f796ef967c0 -1 osd.1 28 log_to_monitors true 2024-06-15T10:59:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T10:59:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T10:59:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T10:59:55.898 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T10:59:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T10:59:55.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T10:59:56.252 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2697747323,v1:127.0.0.1:6811/2697747323] [v2:127.0.0.1:6812/2697747323,v1:127.0.0.1:6813/2697747323] exists,up 195879a4-0386-46c8-a916-79b7d954c749 2024-06-15T10:59:56.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T10:59:56.252 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T10:59:56.253 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T10:59:56.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T10:59:56.254 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T10:59:56.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T10:59:56.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T10:59:56.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T10:59:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T10:59:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T10:59:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T10:59:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T10:59:56.383 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T10:59:56.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T10:59:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T10:59:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T10:59:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T10:59:56.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:56.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T10:59:56.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-06-15T10:59:56.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-06-15T10:59:56.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-06-15T10:59:56.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T10:59:56.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T10:59:57.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-06-15T10:59:57.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-06-15T10:59:57.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-141733920770' 2024-06-15T10:59:57.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:57.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:57.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-06-15T10:59:57.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T10:59:57.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-06-15T10:59:57.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:57.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-06-15T10:59:57.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-06-15T10:59:57.057 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 21474836500 2024-06-15T10:59:57.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:57.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-06-15T10:59:57.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T10:59:58.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T10:59:58.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T10:59:58.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-06-15T10:59:58.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T10:59:58.534 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920770 2024-06-15T10:59:58.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T10:59:58.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T10:59:58.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920770 2024-06-15T10:59:58.538 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T10:59:58.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-06-15T10:59:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2024-06-15T10:59:58.540 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 141733920770 2024-06-15T10:59:58.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T10:59:58.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920770 -lt 141733920770 2024-06-15T10:59:58.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T10:59:58.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:58.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:59.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T10:59:59.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T10:59:59.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T10:59:59.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T10:59:59.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T10:59:59.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T10:59:59.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-06-15T10:59:59.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-06-15T10:59:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-06-15T10:59:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-06-15T10:59:59.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T10:59:59.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T10:59:59.709 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T10:59:59.710 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T10:59:59.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T10:59:59.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T10:59:59.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:00:00.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-06-15T11:00:02.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-06-15T11:00:02.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:00:02.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:00:02.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:00:02.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:00:02.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:00:02.314 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:00:02.314 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:02.314 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:02.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:02.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:00:02.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:00:02.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:00:02.316 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T11:00:02.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:00:02.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:00:02.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:00:02.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:00:02.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:00:02.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:00:02.349 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:02.348+0000 7f4ae20067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:02.353 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:02.352+0000 7f4ae20067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:02.354 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:02.352+0000 7f4ae20067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:02.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:00:02.556 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:00:02.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:02.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:02.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:02.890 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:02.888+0000 7f4ae20067c0 -1 Falling back to public interface 2024-06-15T11:00:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:00:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:00:03.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:03.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:04.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:05.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:00:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:00:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:05.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:05.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:05.448 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:05.448+0000 7f4ae20067c0 -1 osd.0 35 log_to_monitors true 2024-06-15T11:00:06.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:06.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:06.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:00:06.298 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:00:06.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:06.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:06.558 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/746350772,v1:127.0.0.1:6803/746350772] [v2:127.0.0.1:6804/746350772,v1:127.0.0.1:6805/746350772] exists,up bacb953a-786c-41cf-80f0-a760c0793fde 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:00:06.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:00:06.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:00:06.560 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:00:06.560 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:00:06.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:00:06.560 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:00:06.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:00:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:00:06.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:00:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:00:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:00:06.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:00:06.681 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:00:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:00:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:00:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:00:06.914 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:00:06.914 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:00:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-06-15T11:00:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-06-15T11:00:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546' 2024-06-15T11:00:07.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:00:07.062 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:00:07.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-06-15T11:00:07.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-06-15T11:00:07.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-141733920773' 2024-06-15T11:00:07.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:00:07.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724546 2024-06-15T11:00:07.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:00:07.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:00:07.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724546 2024-06-15T11:00:07.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:00:07.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-06-15T11:00:07.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-06-15T11:00:07.203 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 167503724546 2024-06-15T11:00:07.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:00:07.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-06-15T11:00:07.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:00:08.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:00:08.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:00:08.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-06-15T11:00:08.687 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:00:08.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:00:08.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920773 2024-06-15T11:00:08.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:00:08.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920773 2024-06-15T11:00:08.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:00:08.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-06-15T11:00:08.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2024-06-15T11:00:08.693 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 141733920773 2024-06-15T11:00:08.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:00:08.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920773 -lt 141733920773 2024-06-15T11:00:08.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:00:08.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:00:08.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:00:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-06-15T11:00:09.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:00:09.208 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:00:09.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-06-15T11:00:09.441 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:00:09.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:00:09.442 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:00:09.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-06-15T11:00:09.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:00:09.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:00:09.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-06-15T11:00:09.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-06-15T11:00:09.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-06-15T11:00:09.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-06-15T11:00:09.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T11:00:09.700 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:00:09.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:00:09.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T11:00:09.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:09.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-06-15T11:00:10.143 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 3.0 on osd.1 to scrub 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:00:10.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:00:10.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T11:00:10.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T11:00:10.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:00:10.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:00:10.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T11:00:10.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:59:41.648267+0000 '>' 2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:10.380 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:00:11.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:00:11.382 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:00:11.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T11:00:11.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T11:00:11.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:00:11.383 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:00:11.384 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T11:00:11.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T10:59:41.648267+0000 '>' 2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:11.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:00:12.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:00:12.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:00:12.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-06-15T11:00:12.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-06-15T11:00:12.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:00:12.613 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:00:12.614 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-06-15T11:00:12.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:00:10.225011+0000 '>' 2024-06-15T10:59:41.648267+0000 2024-06-15T11:00:12.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:00:12.851 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-06-15T11:00:12.882 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-06-15T11:00:12.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-06-15T11:00:12.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-06-15T11:00:12.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-06-15T11:00:12.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T11:00:12.966 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-15T11:00:12.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T11:00:12.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-06-15T11:00:12.999 INFO:tasks.workunit.client.0.smithi116.stderr: . as $in 2024-06-15T11:00:12.999 INFO:tasks.workunit.client.0.smithi116.stderr: | if type == "object" then 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr: reduce keys[] as $key 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr: elif type == "array" then map( walk(f) ) | f 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr: else f 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr: end; 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.version) else . end) 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-06-15T11:00:13.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-06-15T11:00:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T11:00:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-06-15T11:00:13.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-06-15T11:00:13.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-06-15T11:00:13.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-06-15T11:00:13.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-06-15T11:00:13.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-06-15T11:00:13.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-06-15T11:00:13.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-06-15T11:00:13.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-06-15T11:00:13.358 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'csr_pool' removed 2024-06-15T11:00:13.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T11:00:13.420 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:00:13.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:00:13.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:00:13.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:00:13.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:00:13.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:00:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:00:13.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:00:13.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:00:13.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:00:13.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:00:13.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:00:13.701 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:00:13.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:00:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:00:13.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:00:13.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:00:13.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:13.722 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:13.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:00:13.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:00:13.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:00:13.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:00:13.726 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:00:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:00:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:00:13.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:00:13.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:00:13.729 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:00:13.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:00:13.730 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:00:13.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:00:13.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:00:13.732 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:00:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:00:13.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:00:13.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:00:13.734 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:00:13.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:00:13.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:00:13.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:00:13.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:00:13.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:13.739 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:13.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:00:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:00:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:00:13.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T11:00:13.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T11:00:13.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:13.742 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:13.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T11:00:13.744 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T11:00:13.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T11:00:13.745 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T11:00:13.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T11:00:13.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T11:00:13.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T11:00:13.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T11:00:14.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T11:00:14.127 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:00:14.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:00:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:00:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:14.128 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:14.128 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:14.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T11:00:14.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:00:14.341 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:00:14.342 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:00:14.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:00:14.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:14.342 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:14.342 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:00:14.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:00:14.343 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T11:00:14.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T11:00:14.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:00:14.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:00:14.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T11:00:14.461 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T11:00:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:00:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:00:14.461 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:00:14.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T11:00:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T11:00:15.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T11:00:15.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:00:15.035 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:00:15.036 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:00:15.036 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:15.036 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:15.036 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:15.037 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:00:15.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:00:15.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:00:15.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:15.075 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:15.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:00:15.077 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4106b129-9152-4fff-b711-7a9eb2d3d64a 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 4106b129-9152-4fff-b711-7a9eb2d3d64a 2024-06-15T11:00:15.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4106b129-9152-4fff-b711-7a9eb2d3d64a' 2024-06-15T11:00:15.079 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:00:15.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/dG1mF+LTBRAAaAluRfenD4rGs2ha2u8OQg== 2024-06-15T11:00:15.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/dG1mF+LTBRAAaAluRfenD4rGs2ha2u8OQg=="}' 2024-06-15T11:00:15.095 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4106b129-9152-4fff-b711-7a9eb2d3d64a -i td/osd-scrub-repair/0/new.json 2024-06-15T11:00:15.341 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:00:15.357 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T11:00:15.358 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/dG1mF+LTBRAAaAluRfenD4rGs2ha2u8OQg== --osd-uuid 4106b129-9152-4fff-b711-7a9eb2d3d64a 2024-06-15T11:00:15.392 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.392+0000 7f87a497a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:15.394 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.392+0000 7f87a497a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:15.395 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.392+0000 7f87a497a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:15.396 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.396+0000 7f87a497a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:00:15.396 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.396+0000 7f87a497a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:00:15.396 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.396+0000 7f87a497a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:00:15.483 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:15.480+0000 7f87a497a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T11:00:21.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T11:00:21.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:00:21.102 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T11:00:21.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T11:00:21.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:00:21.373 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:00:21.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T11:00:21.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:00:21.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:00:21.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:00:21.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:00:21.401 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:21.399+0000 7f89a3a2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:21.421 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:21.419+0000 7f89a3a2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:21.423 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:21.419+0000 7f89a3a2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:21.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T11:00:21.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:00:21.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:00:21.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:00:21.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:00:21.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:21.601 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:00:21.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:00:21.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:21.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:21.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:21.982 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:21.979+0000 7f89a3a2d7c0 -1 Falling back to public interface 2024-06-15T11:00:22.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:22.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:22.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:00:22.838 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:00:22.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:22.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:23.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:24.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:24.082 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:24.083 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:00:24.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:00:24.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:24.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:24.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:24.576 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:24.575+0000 7f89a3a2d7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T11:00:25.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:25.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:25.321 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:00:25.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:00:25.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:25.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:25.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:26.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:26.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:26.581 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:00:26.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:00:26.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:26.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:26.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:27.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:27.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:27.823 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:00:27.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:00:27.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:27.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:28.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:29.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:29.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:29.144 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:00:29.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:00:29.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:29.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:29.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:30.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:30.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:30.462 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:00:30.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:00:30.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:30.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:30.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:31.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:31.707 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:31.707 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:00:31.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:00:31.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:31.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:31.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:32.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:32.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:32.944 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:00:32.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:00:32.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:32.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:33.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:34.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:34.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:34.332 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:00:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:00:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:34.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:34.574 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:35.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:35.577 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:00:35.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:00:35.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:35.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:35.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:36.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:36.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:36.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:00:36.821 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:00:36.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:36.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:37.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:38.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:38.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:38.310 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:00:38.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:00:38.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:38.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:38.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:39.648 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:00:39.648 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:00:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:39.649 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:40.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:41.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:41.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:41.315 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:00:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:00:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:41.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:41.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:42.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:42.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:42.713 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:00:42.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:00:42.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:42.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:43.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:44.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:44.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:44.195 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:00:44.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:00:44.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:44.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:44.868 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:45.870 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:00:45.871 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:00:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:45.871 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:46.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:47.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:47.183 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:00:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:00:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:47.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:00:47.426 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/232423550,v1:127.0.0.1:6803/232423550] [v2:127.0.0.1:6804/232423550,v1:127.0.0.1:6805/232423550] exists,up 4106b129-9152-4fff-b711-7a9eb2d3d64a 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T11:00:47.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:00:47.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:00:47.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:47.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:00:47.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:00:47.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:00:47.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f552c9d-945e-4d46-a4cd-3031c192fe1b 2024-06-15T11:00:47.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4f552c9d-945e-4d46-a4cd-3031c192fe1b' 2024-06-15T11:00:47.434 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 4f552c9d-945e-4d46-a4cd-3031c192fe1b 2024-06-15T11:00:47.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:00:47.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfdG1mDUVGGxAArVUh+/Z5CH2Kdhxa2Zk9JQ== 2024-06-15T11:00:47.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfdG1mDUVGGxAArVUh+/Z5CH2Kdhxa2Zk9JQ=="}' 2024-06-15T11:00:47.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f552c9d-945e-4d46-a4cd-3031c192fe1b -i td/osd-scrub-repair/1/new.json 2024-06-15T11:00:47.770 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:00:47.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T11:00:47.791 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfdG1mDUVGGxAArVUh+/Z5CH2Kdhxa2Zk9JQ== --osd-uuid 4f552c9d-945e-4d46-a4cd-3031c192fe1b 2024-06-15T11:00:47.825 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.823+0000 7fc6c34347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:47.846 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.843+0000 7fc6c34347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.847+0000 7fc6c34347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.847+0000 7fc6c34347c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:00:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.847+0000 7fc6c34347c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:00:47.848 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.847+0000 7fc6c34347c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:00:47.957 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:47.955+0000 7fc6c34347c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T11:00:54.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T11:00:54.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:00:54.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T11:00:54.312 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T11:00:54.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:00:54.586 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:00:54.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T11:00:54.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:00:54.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:00:54.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:00:54.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:00:54.618 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:54.615+0000 7f4e3920f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:54.621 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:54.619+0000 7f4e3920f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:54.623 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:54.619+0000 7f4e3920f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:00:54.823 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:00:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:54.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:00:55.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:55.166 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:55.163+0000 7f4e3920f7c0 -1 Falling back to public interface 2024-06-15T11:00:56.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:56.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:56.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:00:56.066 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:00:56.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:56.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:00:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:57.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:57.343 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:57.343 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:00:57.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:00:57.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:57.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:00:57.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:57.796 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:00:57.799+0000 7f4e3920f7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T11:00:58.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:58.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:58.581 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:00:58.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:00:58.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:58.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:00:58.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:00:59.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:00:59.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:00:59.890 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:00:59.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:00:59.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:00:59.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:00.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:01.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:01.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:01.133 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:01:01.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:01:01.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:01.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:01.400 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:02.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:02.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:02.402 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:01:02.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:01:02.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:02.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:02.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:03.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:03.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:03.723 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:01:03.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:01:03.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:03.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:03.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:04.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:04.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:04.967 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:01:04.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:01:04.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:04.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:05.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:06.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:06.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:06.211 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:01:06.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:01:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:06.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:07.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:07.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:07.553 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:01:07.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:01:07.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:07.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:07.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:08.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:08.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:08.892 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:01:08.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:01:08.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:08.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:09.134 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:10.136 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:10.136 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:01:10.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:01:10.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:10.137 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:10.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:11.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:11.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:11.528 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:01:11.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:01:11.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:11.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:11.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:12.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:12.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:12.933 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:01:12.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:01:12.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:12.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:13.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:14.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:14.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:14.653 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:01:14.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:01:14.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:14.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:15.184 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:16.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:16.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:16.186 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:01:16.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:01:16.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:16.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:16.705 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:17.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:17.708 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:17.708 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:01:17.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:01:17.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:17.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:18.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:19.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:19.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:19.374 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:01:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:01:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:19.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:19.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:20.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:20.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:20.764 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:01:20.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:01:20.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:20.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:01:21.016 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3064647727,v1:127.0.0.1:6811/3064647727] [v2:127.0.0.1:6812/3064647727,v1:127.0.0.1:6813/3064647727] exists,up 4f552c9d-945e-4d46-a4cd-3031c192fe1b 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T11:01:21.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:01:21.020 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:01:21.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:01:21.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T11:01:21.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:01:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e2c2078-9e70-411b-bb04-e35ff3c55563 2024-06-15T11:01:21.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2e2c2078-9e70-411b-bb04-e35ff3c55563' 2024-06-15T11:01:21.024 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 2e2c2078-9e70-411b-bb04-e35ff3c55563 2024-06-15T11:01:21.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:01:21.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBdG1mvOTeAhAANp8ie5kVxe7yEL6dZ61ikg== 2024-06-15T11:01:21.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBdG1mvOTeAhAANp8ie5kVxe7yEL6dZ61ikg=="}' 2024-06-15T11:01:21.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e2c2078-9e70-411b-bb04-e35ff3c55563 -i td/osd-scrub-repair/2/new.json 2024-06-15T11:01:21.328 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:01:21.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T11:01:21.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBdG1mvOTeAhAANp8ie5kVxe7yEL6dZ61ikg== --osd-uuid 2e2c2078-9e70-411b-bb04-e35ff3c55563 2024-06-15T11:01:21.379 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.378+0000 7fe2ea5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:21.412 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.414+0000 7fe2ea5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:21.413 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.414+0000 7fe2ea5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:21.414 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.414+0000 7fe2ea5ea7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:01:21.414 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.414+0000 7fe2ea5ea7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:01:21.414 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.414+0000 7fe2ea5ea7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:01:21.483 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:21.482+0000 7fe2ea5ea7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T11:01:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T11:01:27.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:01:27.726 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T11:01:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T11:01:27.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:01:27.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T11:01:27.997 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T11:01:27.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:01:27.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:01:27.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:01:27.998 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:01:28.030 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:28.030+0000 7f4120ef97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:28.032 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:28.034+0000 7f4120ef97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:28.033 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:28.034+0000 7f4120ef97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:01:28.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T11:01:28.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:01:28.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T11:01:28.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:01:28.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:01:28.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:28.254 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:01:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:01:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:28.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:28.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:28.581 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:28.582+0000 7f4120ef97c0 -1 Falling back to public interface 2024-06-15T11:01:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:29.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:29.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:01:29.489 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:01:29.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:29.489 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:29.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:30.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:30.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:30.729 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:01:30.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:01:30.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:30.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:30.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:31.091 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:01:31.090+0000 7f4120ef97c0 -1 osd.2 0 log_to_monitors true 2024-06-15T11:01:31.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:31.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:31.974 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:01:31.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:01:31.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:31.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:32.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:33.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:33.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:33.220 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:01:33.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:01:33.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:33.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:33.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:34.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:34.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:34.545 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:01:34.545 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:01:34.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:34.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:34.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:35.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:35.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:35.787 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:01:35.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:01:35.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:35.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:36.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:37.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:37.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:37.066 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:01:37.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:01:37.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:37.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:37.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:38.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:38.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:38.309 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:01:38.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:01:38.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:38.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:38.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:39.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:39.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:39.551 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:01:39.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:01:39.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:39.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:39.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:40.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:40.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:40.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:01:40.792 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:01:40.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:40.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:41.149 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:42.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:42.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:42.151 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:01:42.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:01:42.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:42.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:42.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:43.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:43.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:43.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:01:43.474 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:01:43.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:43.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:43.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:44.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:44.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:44.718 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:01:44.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:01:44.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:44.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:45.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:46.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:46.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:46.245 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:01:46.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:01:46.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:46.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:46.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:47.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:47.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:47.894 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:01:47.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:01:47.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:47.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:48.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:49.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:49.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:49.456 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:01:49.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:01:49.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:49.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:50.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:51.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:51.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:51.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:01:51.228 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:01:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:51.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:52.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:53.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:53.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:53.016 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:01:53.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:01:53.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:53.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:53.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:01:54.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:01:54.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:01:54.498 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:01:54.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:01:54.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:01:54.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:01:54.738 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1173565326,v1:127.0.0.1:6819/1173565326] [v2:127.0.0.1:6820/1173565326,v1:127.0.0.1:6821/1173565326] exists,up 2e2c2078-9e70-411b-bb04-e35ff3c55563 2024-06-15T11:01:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:01:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:01:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:01:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-15T11:01:54.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T11:01:54.978 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T11:01:54.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T11:01:54.993 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T11:01:55.356 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T11:01:55.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:01:56.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T11:01:58.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-15T11:01:58.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:01:58.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:01:58.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:01:58.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:01:58.090 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:01:58.090 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:01:58.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:01:58.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:01:58.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:01:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:01:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:01:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:01:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:01:58.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:01:58.230 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:01:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:01:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:01:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:01:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:01:58.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:01:58.470 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:01:58.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-15T11:01:58.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-15T11:01:58.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-15T11:01:58.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:01:58.636 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:01:58.775 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-06-15T11:01:58.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-06-15T11:01:58.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672979' 2024-06-15T11:01:58.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:01:58.776 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:01:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-06-15T11:01:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-06-15T11:01:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672979 2-64424509450' 2024-06-15T11:01:58.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:01:58.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-15T11:01:58.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:01:58.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:01:58.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-15T11:01:58.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:01:58.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-15T11:01:58.921 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836497 2024-06-15T11:01:58.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-15T11:01:58.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:01:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-06-15T11:01:59.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:02:00.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:02:00.149 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:00.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-15T11:02:00.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:00.419 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-06-15T11:02:00.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:00.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:02:00.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-06-15T11:02:00.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:00.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-06-15T11:02:00.424 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672979 2024-06-15T11:02:00.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-06-15T11:02:00.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:02:00.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-06-15T11:02:00.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:00.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-06-15T11:02:00.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:00.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:02:00.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-06-15T11:02:00.690 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:00.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-06-15T11:02:00.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-06-15T11:02:00.692 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509450 2024-06-15T11:02:00.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:02:00.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-06-15T11:02:00.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:02:00.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:00.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:02:01.172 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:02:01.386 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T11:02:01.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:02:01.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:01.387 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T11:02:01.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T11:02:01.620 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-15T11:02:01.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T11:02:01.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T11:02:02.611 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T11:02:02.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:02:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T11:02:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T11:02:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:02:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:02:03.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:02:03.637 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:02:03.638 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:02:03.638 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:02:03.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:02:03.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:02:03.639 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:02:03.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:02:03.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:02:03.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:02:03.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:02:03.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:02:03.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:02:04.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:02:04.022 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:04.022 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:02:04.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:02:04.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:04.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:02:04.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-15T11:02:04.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-15T11:02:04.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-15T11:02:04.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:04.170 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:02:04.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-06-15T11:02:04.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-06-15T11:02:04.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672981' 2024-06-15T11:02:04.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:04.321 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:02:04.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-06-15T11:02:04.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-06-15T11:02:04.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672981 2-64424509452' 2024-06-15T11:02:04.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:04.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-15T11:02:04.465 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:04.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:02:04.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-15T11:02:04.468 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:04.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-15T11:02:04.469 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836499 2024-06-15T11:02:04.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-15T11:02:04.469 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:04.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-06-15T11:02:04.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:02:05.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:02:05.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:05.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-06-15T11:02:05.943 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:05.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-06-15T11:02:05.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:05.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:02:05.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-06-15T11:02:05.948 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:05.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-06-15T11:02:05.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-06-15T11:02:05.949 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672981 2024-06-15T11:02:05.949 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:02:06.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-06-15T11:02:06.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:06.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-06-15T11:02:06.197 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:06.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:02:06.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-06-15T11:02:06.200 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:06.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-06-15T11:02:06.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-06-15T11:02:06.202 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509452 2024-06-15T11:02:06.202 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:02:06.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-06-15T11:02:06.455 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:02:06.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:06.456 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:02:06.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:02:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:02:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:02:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:02:06.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:02:06.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:02:06.712 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:02:06.995 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:02:06.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:02:06.996 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:06.997 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:07.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T11:02:07.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T11:02:07.617 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T11:02:07.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T11:02:08.011 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T11:02:08.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T11:02:08.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T11:02:08.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:02:08.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-15T11:02:08.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T11:02:08.226 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-15T11:02:08.227 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-15T11:02:08.228 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T11:02:08.490 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T11:02:08.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T11:02:08.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T11:02:08.856 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T11:02:08.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T11:02:09.359 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T11:02:09.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T11:02:09.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T11:02:09.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:02:09.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-15T11:02:09.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T11:02:09.725 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-15T11:02:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T11:02:09.726 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-15T11:02:09.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-15T11:02:09.973 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:09.973 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T11:02:09.973 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-15T11:02:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-15T11:02:09.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-15T11:02:09.974 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-15T11:02:09.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:09.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T11:02:09.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:02:09.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:02:09.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:02:09.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:02:09.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:02:10.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:02:10.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:10.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T11:02:10.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:02:10.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:02:10.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:02:10.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:02:10.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:02:11.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:02:11.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:11.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T11:02:11.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:02:11.605 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:02:11.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:02:11.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:02:11.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:02:11.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:02:11.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-15T11:02:11.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-15T11:02:13.029 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-15T11:02:13.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-15T11:02:13.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-15T11:02:14.840 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-15T11:02:15.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-15T11:02:15.374 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-15T11:02:16.136 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-15T11:02:16.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-15T11:02:16.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-15T11:02:17.788 INFO:tasks.workunit.client.0.smithi116.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-15T11:02:18.327 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:02:18.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:02:18.330 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:18.331 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:02:18.333 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:02:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:02:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:02:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:02:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:02:18.334 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T11:02:18.335 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:02:18.335 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:02:18.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:02:18.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:02:18.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:02:18.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:02:18.369 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:18.369+0000 7fc71066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:18.372 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:18.373+0000 7fc71066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:18.373 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:18.373+0000 7fc71066d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:02:18.577 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:02:18.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:18.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:18.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:18.926 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:18.925+0000 7fc71066d7c0 -1 Falling back to public interface 2024-06-15T11:02:19.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:19.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:19.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:02:19.830 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:19.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:19.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:20.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:02:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:02:21.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:21.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:21.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:21.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:21.585+0000 7fc71066d7c0 -1 osd.0 33 log_to_monitors true 2024-06-15T11:02:22.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:22.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:22.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:02:22.314 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:02:22.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:22.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/783776009,v1:127.0.0.1:6803/783776009] [v2:127.0.0.1:6804/783776009,v1:127.0.0.1:6805/783776009] exists,up 4106b129-9152-4fff-b711-7a9eb2d3d64a 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-15T11:02:22.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:02:22.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:02:22.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:02:22.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:02:22.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:02:22.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:22.604 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:22.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T11:02:22.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:02:22.608 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T11:02:22.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T11:02:22.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:02:22.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:02:22.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:02:22.637 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:22.637+0000 7f8da00647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:22.682 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:22.681+0000 7f8da00647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:22.683 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:22.685+0000 7f8da00647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:02:22.872 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:02:22.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:22.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:02:23.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:23.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:23.237+0000 7f8da00647c0 -1 Falling back to public interface 2024-06-15T11:02:24.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:24.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:24.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:02:24.116 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:24.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:24.117 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:02:24.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:02:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:02:25.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:25.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:02:25.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:25.862 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:25.861+0000 7f8da00647c0 -1 osd.1 35 log_to_monitors true 2024-06-15T11:02:26.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:26.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:26.605 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:02:26.605 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:02:26.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:26.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1637521150,v1:127.0.0.1:6811/1637521150] [v2:127.0.0.1:6812/1637521150,v1:127.0.0.1:6813/1637521150] exists,up 4f552c9d-945e-4d46-a4cd-3031c192fe1b 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:02:26.860 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:02:26.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:02:26.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:02:26.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T11:02:26.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T11:02:26.864 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T11:02:26.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:02:26.865 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T11:02:26.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T11:02:26.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:02:26.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:02:26.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:02:26.888 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:26.889+0000 7fa089c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:26.891 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:26.889+0000 7fa089c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:26.892 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:26.893+0000 7fa089c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:27.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T11:02:27.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:02:27.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:27.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:02:27.442 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:27.441+0000 7fa089c9b7c0 -1 Falling back to public interface 2024-06-15T11:02:27.447 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:28.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:02:28.688 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:02:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:02:29.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:29.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:02:29.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:30.000 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:30.001+0000 7fa089c9b7c0 -1 osd.2 37 log_to_monitors true 2024-06-15T11:02:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:30.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:02:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:02:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:30.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:02:31.292 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/2050081586,v1:127.0.0.1:6819/2050081586] [v2:127.0.0.1:6820/2050081586,v1:127.0.0.1:6821/2050081586] exists,up 2e2c2078-9e70-411b-bb04-e35ff3c55563 2024-06-15T11:02:31.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:02:31.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:02:31.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:02:31.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-15T11:02:31.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T11:02:31.866 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' removed 2024-06-15T11:02:31.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T11:02:31.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T11:02:32.411 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T11:02:32.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:02:33.437 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T11:02:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-15T11:02:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:02:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:02:34.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:02:34.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:02:34.960 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:02:34.960 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:02:34.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:02:34.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:02:34.961 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:02:35.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:02:35.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:02:35.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:02:35.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:02:35.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:02:35.100 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:02:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:02:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:02:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:02:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:02:35.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:35.340 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:02:35.487 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-06-15T11:02:35.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-06-15T11:02:35.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-06-15T11:02:35.488 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:35.488 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:02:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-06-15T11:02:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-06-15T11:02:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-06-15T11:02:35.629 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:02:35.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:02:35.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-06-15T11:02:35.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-06-15T11:02:35.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-06-15T11:02:35.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:35.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-06-15T11:02:35.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:35.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:02:35.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-06-15T11:02:35.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:35.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-06-15T11:02:35.777 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 171798691844 2024-06-15T11:02:35.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-06-15T11:02:35.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:36.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-06-15T11:02:36.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:02:37.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:02:37.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:37.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-06-15T11:02:37.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:02:38.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T11:02:38.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:02:38.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691844 2024-06-15T11:02:38.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:38.496 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-06-15T11:02:38.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:38.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:02:38.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-06-15T11:02:38.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:38.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-06-15T11:02:38.501 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 184683593731 2024-06-15T11:02:38.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-06-15T11:02:38.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:02:38.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593731 2024-06-15T11:02:38.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:02:38.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:02:38.748 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-06-15T11:02:38.749 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:02:38.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-06-15T11:02:38.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:02:38.751 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-06-15T11:02:38.751 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 201863462915 2024-06-15T11:02:38.752 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-06-15T11:02:38.752 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:02:39.003 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-06-15T11:02:39.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:02:39.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:39.004 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:39.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:02:39.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:02:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:02:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:02:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:02:39.286 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:02:39.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:02:39.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:02:39.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:02:39.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:02:39.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:02:39.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:02:39.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:02:39.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:02:39.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:02:39.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-15T11:02:39.807 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T11:02:39.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-15T11:02:39.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-15T11:02:39.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T11:02:40.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-15T11:02:40.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-15T11:02:40.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T11:02:40.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T11:02:40.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:40.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:40.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:40.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:40.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:40.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T11:02:40.498 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:02:40.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:02:40.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:02:40.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:40.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:40.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:40.516 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:40.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:02:02.453868+0000 '>' 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:40.746 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:02:41.747 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:02:41.748 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:02:41.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:02:41.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:41.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:41.749 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:41.750 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:41.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:02:02.453868+0000 '>' 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:41.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:02:42.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:02:42.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:02:42.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:02:42.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:42.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:42.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:42.995 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:43.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:02:02.453868+0000 '>' 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:43.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:02:44.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:02:44.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:02:44.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:02:44.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:44.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:44.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:44.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:44.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:02:02.453868+0000 '>' 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:44.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:02:45.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:02:45.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:02:45.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:02:45.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:02:45.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:02:45.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:02:45.477 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:02:45.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:02:41.071118+0000 '>' 2024-06-15T11:02:02.453868+0000 2024-06-15T11:02:45.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:02:45.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-15T11:02:45.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-15T11:02:45.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-15T11:02:45.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-15T11:02:45.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-15T11:02:45.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-15T11:02:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-15T11:02:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-15T11:02:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-15T11:02:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T11:02:45.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:02:45.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:02:45.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:02:46.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:02:46.219 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:02:46.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:02:46.221 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:02:46.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:02:46.223 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:02:46.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:02:46.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:02:46.225 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:02:46.226 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:02:46.227 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:02:46.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:02:46.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:02:46.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:02:46.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:02:46.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:46.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:46.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:02:46.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:02:46.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:02:46.257 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:02:46.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:02:46.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:02:46.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:02:46.260 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:02:46.261 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:02:46.262 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:02:46.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:02:46.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:02:46.264 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:02:46.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:02:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:02:46.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:02:46.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:02:46.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:02:46.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:02:46.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:02:46.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:02:46.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:02:46.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:46.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:46.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:02:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:02:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:02:46.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T11:02:46.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T11:02:46.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:46.275 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:46.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T11:02:46.277 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T11:02:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T11:02:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T11:02:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T11:02:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-06-15T11:02:46.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T11:02:46.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:46.674 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:46.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:46.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T11:02:46.886 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T11:02:46.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T11:02:46.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:02:46.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:02:46.887 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T11:02:46.888 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T11:02:46.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:02:46.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:02:46.888 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:02:46.889 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:02:46.889 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:46.889 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:46.889 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:02:46.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:02:46.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T11:02:47.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T11:02:47.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:02:47.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:02:47.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:47.008 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:47.009 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:02:47.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:02:47.009 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T11:02:47.119 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T11:02:47.120 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T11:02:47.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T11:02:47.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:47.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:47.976 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:47.976 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:47.976 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:47.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:47.977 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:02:47.979 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:02:48.012 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:02:48.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:02:48.016 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:02:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:02:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:02:48.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:02:48.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3c1473b-47db-4629-9a2a-598a4896ecfa 2024-06-15T11:02:48.019 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 b3c1473b-47db-4629-9a2a-598a4896ecfa 2024-06-15T11:02:48.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b3c1473b-47db-4629-9a2a-598a4896ecfa' 2024-06-15T11:02:48.019 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:02:48.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYdG1mvXBKAhAAuBI9gHXi6PpCssctaazjBw== 2024-06-15T11:02:48.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYdG1mvXBKAhAAuBI9gHXi6PpCssctaazjBw=="}' 2024-06-15T11:02:48.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3c1473b-47db-4629-9a2a-598a4896ecfa -i td/osd-scrub-repair/0/new.json 2024-06-15T11:02:48.345 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:02:48.361 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T11:02:48.362 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYdG1mvXBKAhAAuBI9gHXi6PpCssctaazjBw== --osd-uuid b3c1473b-47db-4629-9a2a-598a4896ecfa 2024-06-15T11:02:48.390 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.389+0000 7f78b4dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:48.392 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.393+0000 7f78b4dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:48.393 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.393+0000 7f78b4dd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:48.393 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.393+0000 7f78b4dd17c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:02:48.393 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.393+0000 7f78b4dd17c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:02:48.393 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.393+0000 7f78b4dd17c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:02:48.471 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:48.469+0000 7f78b4dd17c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T11:02:54.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T11:02:54.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:02:54.079 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T11:02:54.080 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T11:02:54.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:02:54.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T11:02:54.347 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:02:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:02:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:02:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:02:54.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:02:54.378 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:54.377+0000 7ff5b7de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:54.380 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:54.381+0000 7ff5b7de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:54.381 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:54.381+0000 7ff5b7de97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:02:54.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T11:02:54.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:02:54.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:02:54.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:54.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:54.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:54.914 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:54.913+0000 7ff5b7de97c0 -1 Falling back to public interface 2024-06-15T11:02:55.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:55.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:55.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:02:55.820 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:02:55.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:55.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:56.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:57.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:57.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:57.063 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:02:57.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:02:57.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:57.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:57.305 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:57.485 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:02:57.485+0000 7ff5b7de97c0 -1 osd.0 0 log_to_monitors true 2024-06-15T11:02:58.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:58.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:58.307 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:02:58.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:02:58.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:58.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:58.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:02:59.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:02:59.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:02:59.612 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:02:59.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:02:59.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:02:59.613 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:02:59.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:00.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:00.844 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:00.845 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:03:00.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:03:00.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:00.845 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:01.112 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:02.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:02.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:02.115 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:03:02.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:03:02.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:02.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:02.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:03.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:03.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:03.416 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:03:03.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:03:03.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:03.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:03.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:04.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:04.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:04.661 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:03:04.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:03:04.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:04.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:04.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:05.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:05.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:05.904 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:03:05.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:03:05.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:05.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:06.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:07.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:07.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:07.159 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:03:07.159 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:03:07.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:07.160 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:07.453 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:08.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:08.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:08.455 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:03:08.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:03:08.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:08.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:08.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:09.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:09.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:09.697 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:03:09.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:03:09.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:09.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:10.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:11.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:11.181 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:11.182 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:03:11.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:03:11.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:11.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:11.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:12.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:12.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:12.645 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:03:12.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:03:12.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:12.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:13.040 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:14.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:14.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:14.042 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:03:14.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:03:14.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:14.043 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:14.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:15.699 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:15.700 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:03:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:03:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:15.700 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:16.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:17.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:17.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:17.430 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:03:17.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:03:17.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:17.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:18.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:19.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:19.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:19.213 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:03:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:03:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:19.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:19.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:20.464 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:03:20.465 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:03:20.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:20.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:03:20.709 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3625129298,v1:127.0.0.1:6803/3625129298] [v2:127.0.0.1:6804/3625129298,v1:127.0.0.1:6805/3625129298] exists,up b3c1473b-47db-4629-9a2a-598a4896ecfa 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:03:20.711 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:03:20.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:03:20.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:03:20.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:03:20.713 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:03:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:03:20.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:03:20.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:03:20.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f6b4062a-3756-4480-b547-1f042c62eb05 2024-06-15T11:03:20.716 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 f6b4062a-3756-4480-b547-1f042c62eb05 2024-06-15T11:03:20.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f6b4062a-3756-4480-b547-1f042c62eb05' 2024-06-15T11:03:20.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:03:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4dG1mLMHuKxAAIswGCWEnI7+reVL7WarbIg== 2024-06-15T11:03:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4dG1mLMHuKxAAIswGCWEnI7+reVL7WarbIg=="}' 2024-06-15T11:03:20.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f6b4062a-3756-4480-b547-1f042c62eb05 -i td/osd-scrub-repair/1/new.json 2024-06-15T11:03:21.020 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:03:21.037 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T11:03:21.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4dG1mLMHuKxAAIswGCWEnI7+reVL7WarbIg== --osd-uuid f6b4062a-3756-4480-b547-1f042c62eb05 2024-06-15T11:03:21.074 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.072+0000 7fb0e5f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:21.105 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.104+0000 7fb0e5f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:21.106 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.104+0000 7fb0e5f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.104+0000 7fb0e5f857c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:03:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.104+0000 7fb0e5f857c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:03:21.107 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.104+0000 7fb0e5f857c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:03:21.175 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:21.172+0000 7fb0e5f857c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T11:03:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T11:03:27.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:03:27.289 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T11:03:27.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T11:03:27.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:03:27.566 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:03:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T11:03:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:03:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:03:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:03:27.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:03:27.598 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:27.596+0000 7f06295537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:27.626 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:27.624+0000 7f06295537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:27.627 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:27.624+0000 7f06295537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:27.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T11:03:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:03:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:03:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:03:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:03:27.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:27.810 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:03:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:03:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:27.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:28.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:28.190 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:28.188+0000 7f06295537c0 -1 Falling back to public interface 2024-06-15T11:03:29.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:29.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:29.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:03:29.058 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:03:29.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:29.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:29.291 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:30.293 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:03:30.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:03:30.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:30.294 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:30.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:30.811 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:30.808+0000 7f06295537c0 -1 osd.1 0 log_to_monitors true 2024-06-15T11:03:31.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:31.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:31.540 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:03:31.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:03:31.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:31.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:31.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:32.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:32.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:32.903 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:03:32.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:03:32.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:32.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:33.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:34.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:34.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:34.155 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:03:34.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:03:34.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:34.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:34.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:35.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:35.407 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:03:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:03:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:35.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:35.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:36.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:36.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:36.704 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:03:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:03:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:36.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:37.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:37.950 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:37.950 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:03:37.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:03:37.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:37.951 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:38.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:39.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:39.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:39.185 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:03:39.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:03:39.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:39.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:39.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:40.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:40.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:40.442 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:03:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:03:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:40.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:40.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:41.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:41.761 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:03:41.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:41.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:41.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:43.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:43.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:43.001 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:03:43.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:03:43.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:43.001 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:44.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:44.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:44.303 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:03:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:03:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:44.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:44.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:45.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:45.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:45.948 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:03:45.949 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:03:45.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:45.949 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:46.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:47.210 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:47.211 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:03:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:03:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:47.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:47.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:48.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:48.835 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:48.835 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:03:48.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:03:48.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:48.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:49.201 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:50.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:50.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:50.203 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:03:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:03:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:50.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:50.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:51.663 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:51.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:51.664 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:03:51.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:03:51.664 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:51.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:52.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:53.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:53.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:53.237 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:03:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:03:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:53.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:53.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:03:54.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:03:54.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:03:54.512 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T11:03:54.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T11:03:54.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:03:54.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:03:54.750 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3114940943,v1:127.0.0.1:6811/3114940943] [v2:127.0.0.1:6812/3114940943,v1:127.0.0.1:6813/3114940943] exists,up f6b4062a-3756-4480-b547-1f042c62eb05 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T11:03:54.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:03:54.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:03:54.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:03:54.755 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:03:54.756 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:03:54.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:03:54.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6fbe6fc0-38fe-4552-91c0-51ec09e4738e 2024-06-15T11:03:54.759 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 6fbe6fc0-38fe-4552-91c0-51ec09e4738e 2024-06-15T11:03:54.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6fbe6fc0-38fe-4552-91c0-51ec09e4738e' 2024-06-15T11:03:54.759 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:03:54.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAadW1ms3ioLhAAInicIk+XBIMyvFkfpFA7iA== 2024-06-15T11:03:54.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAadW1ms3ioLhAAInicIk+XBIMyvFkfpFA7iA=="}' 2024-06-15T11:03:54.781 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6fbe6fc0-38fe-4552-91c0-51ec09e4738e -i td/osd-scrub-repair/2/new.json 2024-06-15T11:03:55.067 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:03:55.084 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T11:03:55.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAadW1ms3ioLhAAInicIk+XBIMyvFkfpFA7iA== --osd-uuid 6fbe6fc0-38fe-4552-91c0-51ec09e4738e 2024-06-15T11:03:55.119 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.116+0000 7f42ad6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:55.151 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.148+0000 7f42ad6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:55.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.152+0000 7f42ad6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:03:55.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.152+0000 7f42ad6977c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:03:55.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.152+0000 7f42ad6977c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:03:55.153 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.152+0000 7f42ad6977c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:03:55.227 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:03:55.224+0000 7f42ad6977c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T11:04:01.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T11:04:01.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:04:01.320 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T11:04:01.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T11:04:01.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:04:01.589 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T11:04:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T11:04:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:04:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:04:01.590 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:04:01.591 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:04:01.623 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:01.620+0000 7f3efeb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:01.648 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:01.648+0000 7f3efeb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:01.649 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:01.648+0000 7f3efeb867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:01.847 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:04:01.848 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:04:01.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:01.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:02.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:02.206 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:02.204+0000 7f3efeb867c0 -1 Falling back to public interface 2024-06-15T11:04:03.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:03.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:03.105 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:04:03.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:04:03.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:03.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:03.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:04.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:04.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:04.350 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:04:04.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:04:04.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:04.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:04.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:04.846 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:04.844+0000 7f3efeb867c0 -1 osd.2 0 log_to_monitors true 2024-06-15T11:04:05.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:05.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:05.594 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:04:05.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:04:05.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:05.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:05.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:06.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:06.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:06.969 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:04:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:04:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:06.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:07.348 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:08.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:08.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:08.350 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:04:08.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:04:08.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:08.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:08.738 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:09.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:09.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:09.740 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:04:09.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:04:09.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:09.741 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:09.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:10.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:10.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:10.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:04:10.984 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:04:10.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:10.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:11.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:12.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:12.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:12.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:04:12.355 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:04:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:12.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:12.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:13.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:13.764 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:13.765 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:04:13.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:04:13.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:13.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:14.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:15.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:15.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:15.009 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:04:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:04:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:15.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:15.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:16.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:16.331 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:16.331 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:04:16.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:04:16.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:16.332 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:16.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:17.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:17.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:17.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:04:17.581 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:04:17.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:17.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:18.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:19.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:19.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:19.285 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:04:19.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:04:19.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:19.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:19.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:20.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:20.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:20.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:04:20.988 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:04:20.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:20.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:21.608 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:22.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:22.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:22.611 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:04:22.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:04:22.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:22.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:23.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:24.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:24.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:24.718 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:04:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:04:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:24.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:25.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:26.022 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:26.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:26.023 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:04:26.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:04:26.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:26.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:26.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:27.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:27.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:27.812 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:04:27.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:04:27.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:27.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:28.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:29.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:29.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:29.056 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:04:29.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:04:29.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:29.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:04:29.335 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4189369878,v1:127.0.0.1:6819/4189369878] [v2:127.0.0.1:6820/4189369878,v1:127.0.0.1:6821/4189369878] exists,up 6fbe6fc0-38fe-4552-91c0-51ec09e4738e 2024-06-15T11:04:29.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:04:29.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:04:29.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:04:29.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-06-15T11:04:29.336 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T11:04:29.601 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T11:04:29.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T11:04:29.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T11:04:29.963 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T11:04:29.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:04:30.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T11:04:32.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-06-15T11:04:32.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:04:32.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:04:32.692 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:04:32.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:04:32.694 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:04:32.694 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:04:32.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:04:32.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:04:32.695 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:04:32.832 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:04:33.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:04:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:04:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:04:33.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:33.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:04:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-15T11:04:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-15T11:04:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-15T11:04:33.217 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:33.218 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:04:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-06-15T11:04:33.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-06-15T11:04:33.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672973' 2024-06-15T11:04:33.364 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:33.364 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:04:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542154 2024-06-15T11:04:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542154 2024-06-15T11:04:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672973 2-60129542154' 2024-06-15T11:04:33.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:33.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-15T11:04:33.505 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:33.506 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:04:33.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-15T11:04:33.507 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:33.508 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-15T11:04:33.509 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-15T11:04:33.509 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836497 2024-06-15T11:04:33.509 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:04:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-06-15T11:04:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:04:34.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:04:34.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:04:34.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-15T11:04:34.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:34.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-06-15T11:04:34.969 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:34.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:04:34.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-06-15T11:04:34.972 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:34.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-06-15T11:04:34.974 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672973 2024-06-15T11:04:34.974 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-06-15T11:04:34.975 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:04:35.218 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-06-15T11:04:35.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:35.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:35.220 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542154 2024-06-15T11:04:35.221 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:04:35.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542154 2024-06-15T11:04:35.222 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:35.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542154 2024-06-15T11:04:35.224 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2024-06-15T11:04:35.224 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 60129542154 2024-06-15T11:04:35.224 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:04:35.482 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542154 -lt 60129542154 2024-06-15T11:04:35.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:04:35.483 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:04:35.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:04:35.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T11:04:35.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:04:35.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:04:35.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T11:04:35.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:04:35.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:04:35.968 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-06-15T11:04:36.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T11:04:36.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-15T11:04:36.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-06-15T11:04:36.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-06-15T11:04:37.244 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'ecpool' created 2024-06-15T11:04:37.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:04:38.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-06-15T11:04:38.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-06-15T11:04:38.869 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 2 allow_ec_overwrites to true 2024-06-15T11:04:38.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T11:04:38.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:04:38.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:04:38.892 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:04:38.893 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:04:38.894 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:04:38.894 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:04:38.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:04:38.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:04:38.895 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:04:39.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:04:39.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:04:39.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:04:39.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:04:39.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:04:39.026 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:39.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:04:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-06-15T11:04:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-06-15T11:04:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-06-15T11:04:39.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:39.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:04:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-06-15T11:04:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-06-15T11:04:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2024-06-15T11:04:39.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:04:39.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:04:39.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542156 2024-06-15T11:04:39.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542156 2024-06-15T11:04:39.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-60129542156' 2024-06-15T11:04:39.736 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:39.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-06-15T11:04:39.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:39.737 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:04:39.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-06-15T11:04:39.738 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:39.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-06-15T11:04:39.739 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-06-15T11:04:39.740 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836499 2024-06-15T11:04:39.740 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:04:39.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-06-15T11:04:39.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:04:40.970 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:04:40.970 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:04:41.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836499 2024-06-15T11:04:41.228 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:41.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-06-15T11:04:41.229 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:41.231 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:04:41.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-06-15T11:04:41.233 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-06-15T11:04:41.235 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672976 2024-06-15T11:04:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-06-15T11:04:41.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:04:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-06-15T11:04:41.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:04:41.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542156 2024-06-15T11:04:41.472 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:04:41.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:04:41.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542156 2024-06-15T11:04:41.476 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:04:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542156 2024-06-15T11:04:41.477 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 60129542156 2024-06-15T11:04:41.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2024-06-15T11:04:41.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:04:41.709 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542156 -lt 60129542156 2024-06-15T11:04:41.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:04:41.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:04:41.711 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:04:41.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:04:41.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:04:41.979 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:04:42.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:04:42.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:04:42.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:04:42.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-06-15T11:04:42.469 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T11:04:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T11:04:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-06-15T11:04:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T11:04:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T11:04:42.470 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T11:04:42.802 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T11:04:42.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T11:04:43.321 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T11:04:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T11:04:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T11:04:43.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:04:43.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-06-15T11:04:43.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T11:04:43.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-06-15T11:04:43.552 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-06-15T11:04:43.552 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T11:04:43.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-15T11:04:43.808 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:43.808 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T11:04:43.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-15T11:04:43.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-06-15T11:04:43.809 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-06-15T11:04:44.149 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is set 2024-06-15T11:04:44.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-06-15T11:04:44.567 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T11:04:44.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T11:04:44.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T11:04:44.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:04:44.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-06-15T11:04:44.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-06-15T11:04:44.842 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-06-15T11:04:44.842 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-06-15T11:04:44.842 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-06-15T11:04:45.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-06-15T11:04:45.181 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:45.181 INFO:tasks.workunit.client.0.smithi116.stderr:0' 2024-06-15T11:04:45.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-06-15T11:04:45.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-06-15T11:04:45.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-06-15T11:04:45.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-06-15T11:04:45.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:04:45.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T11:04:45.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:04:45.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:04:45.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:04:45.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:04:45.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:04:45.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:04:45.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:04:45.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T11:04:45.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:04:45.497 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:04:45.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:04:45.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:04:45.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:04:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:04:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:04:46.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-06-15T11:04:46.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:04:46.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:04:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:04:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:04:46.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:04:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:04:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-06-15T11:04:47.127 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-06-15T11:04:48.345 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-06-15T11:04:48.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-06-15T11:04:48.878 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-06-15T11:04:50.031 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-06-15T11:04:50.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-06-15T11:04:50.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-06-15T11:04:51.460 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-06-15T11:04:51.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-06-15T11:04:51.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-06-15T11:04:53.112 INFO:tasks.workunit.client.0.smithi116.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:04:53.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:04:53.652 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:04:53.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:04:53.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:04:53.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:04:53.656 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T11:04:53.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:04:53.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:04:53.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:04:53.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:04:53.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:04:53.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:04:53.694 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:53.695+0000 7f81d6d027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:53.696 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:53.699+0000 7f81d6d027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:53.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:53.699+0000 7f81d6d027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:53.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:04:53.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:54.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:54.234 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:54.235+0000 7f81d6d027c0 -1 Falling back to public interface 2024-06-15T11:04:55.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:55.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:55.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:04:55.147 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:55.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:55.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:04:55.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:04:56.390 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:04:56.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:56.391 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:04:56.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:56.861 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:56.863+0000 7f81d6d027c0 -1 osd.0 33 log_to_monitors true 2024-06-15T11:04:57.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:57.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:57.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:04:57.635 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:04:57.635 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:57.636 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/553006333,v1:127.0.0.1:6803/553006333] [v2:127.0.0.1:6804/553006333,v1:127.0.0.1:6805/553006333] exists,up b3c1473b-47db-4629-9a2a-598a4896ecfa 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:04:57.940 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:04:57.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:04:57.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:04:57.943 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:04:57.943 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:04:57.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:04:57.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T11:04:57.945 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.1 2024-06-15T11:04:57.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:04:57.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T11:04:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T11:04:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:04:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:04:57.946 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:04:57.976 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:57.979+0000 7f76b89087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:57.997 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:57.999+0000 7f76b89087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:57.999 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:57.999+0000 7f76b89087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:04:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T11:04:58.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:58.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:04:58.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:04:58.554 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:04:58.555+0000 7f76b89087c0 -1 Falling back to public interface 2024-06-15T11:04:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:04:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:04:59.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:04:59.467 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:04:59.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:04:59.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:04:59.710 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:00.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:00.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:00.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:05:00.712 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:05:00.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:00.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:05:00.937 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:00.939+0000 7f76b89087c0 -1 osd.1 34 log_to_monitors true 2024-06-15T11:05:00.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:01.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:01.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:01.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:05:01.958 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:05:01.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:01.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:05:02.320 INFO:tasks.workunit.client.0.smithi116.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/3430107989,v1:127.0.0.1:6811/3430107989] [v2:127.0.0.1:6812/3430107989,v1:127.0.0.1:6813/3430107989] exists,up f6b4062a-3756-4480-b547-1f042c62eb05 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:05:02.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:02.322 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:05:02.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.2 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-06-15T11:05:02.325 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-06-15T11:05:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-06-15T11:05:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:05:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:05:02.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:05:02.350 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:02.351+0000 7f56995807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:02.434 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:02.435+0000 7f56995807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:02.435 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:02.435+0000 7f56995807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:02.658 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:05:02.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:02.989 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:02.991+0000 7f56995807c0 -1 Falling back to public interface 2024-06-15T11:05:03.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:03.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:03.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:05:03.900 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:05:03.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:03.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:05:04.142 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:05.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:05:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:05:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:05.144 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:05:05.439 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:05.663 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:05.663+0000 7f56995807c0 -1 osd.2 37 log_to_monitors true 2024-06-15T11:05:06.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:06.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:05:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:05:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:06.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [14,37) [v2:127.0.0.1:6818/614811254,v1:127.0.0.1:6819/614811254] [v2:127.0.0.1:6820/614811254,v1:127.0.0.1:6821/614811254] exists,up 6fbe6fc0-38fe-4552-91c0-51ec09e4738e 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-06-15T11:05:06.849 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T11:05:07.350 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' removed 2024-06-15T11:05:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T11:05:07.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T11:05:07.869 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T11:05:07.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:05:08.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T11:05:10.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-06-15T11:05:10.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:05:10.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:05:10.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:05:10.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:05:10.292 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:05:10.292 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:05:10.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:05:10.292 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:05:10.293 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:05:10.434 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:05:10.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:05:10.672 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:05:10.672 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:05:10.672 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:05:10.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:05:10.673 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:05:10.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-06-15T11:05:10.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-06-15T11:05:10.821 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-06-15T11:05:10.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:05:10.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:05:10.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-06-15T11:05:10.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-06-15T11:05:10.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-06-15T11:05:10.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:05:10.965 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:05:11.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-06-15T11:05:11.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-06-15T11:05:11.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-06-15T11:05:11.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:05:11.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-06-15T11:05:11.107 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:05:11.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:05:11.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-06-15T11:05:11.109 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:05:11.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-06-15T11:05:11.110 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-06-15T11:05:11.110 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 171798691844 2024-06-15T11:05:11.111 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:05:11.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-06-15T11:05:11.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:05:12.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:05:12.323 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:05:12.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691844 2024-06-15T11:05:12.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:05:12.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-06-15T11:05:12.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:05:12.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:05:12.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-06-15T11:05:12.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:05:12.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-06-15T11:05:12.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-06-15T11:05:12.565 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 184683593731 2024-06-15T11:05:12.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:05:12.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593731 2024-06-15T11:05:12.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:05:12.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-06-15T11:05:12.802 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:05:12.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:05:12.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-06-15T11:05:12.804 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:05:12.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-06-15T11:05:12.806 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 201863462915 2024-06-15T11:05:12.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-06-15T11:05:12.806 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:05:13.044 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-06-15T11:05:13.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:05:13.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:05:13.045 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:05:13.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:05:13.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:05:13.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:05:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:05:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:05:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:05:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:05:13.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:05:13.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:05:13.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:05:13.562 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:05:13.563 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:05:13.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:05:13.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:05:13.908 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:05:13.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-06-15T11:05:13.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-06-15T11:05:13.908 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-06-15T11:05:13.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-06-15T11:05:13.909 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T11:05:14.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-06-15T11:05:14.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-06-15T11:05:14.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-06-15T11:05:14.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-06-15T11:05:14.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:05:14.132 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:05:14.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:05:14.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:05:14.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:14.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-06-15T11:05:14.568 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-06-15T11:05:14.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:14.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T11:05:14.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:14.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:05:14.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:05:14.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:05:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:05:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:05:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:05:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:05:14.586 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:05:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:04:37.129798+0000 '>' 2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:14.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:05:15.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:05:15.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:05:15.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:05:15.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:05:15.821 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:05:15.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:05:15.822 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:05:16.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:04:37.129798+0000 '>' 2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:16.061 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:05:17.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:05:17.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:05:17.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:05:17.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:05:17.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:05:17.064 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:05:17.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:05:17.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:05:14.678442+0000 '>' 2024-06-15T11:04:37.129798+0000 2024-06-15T11:05:17.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:05:17.307 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-06-15T11:05:17.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-06-15T11:05:17.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-06-15T11:05:17.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-06-15T11:05:17.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-06-15T11:05:17.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-06-15T11:05:17.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-06-15T11:05:17.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-06-15T11:05:17.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-06-15T11:05:17.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T11:05:17.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:05:17.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:05:17.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:05:17.459 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:05:17.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:05:17.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:05:17.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:05:17.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:05:17.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:05:17.770 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:05:17.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:05:17.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:05:17.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:05:17.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:05:17.775 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:05:17.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:05:17.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:05:17.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:05:17.779 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:05:17.780 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:05:17.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:05:17.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:17.808 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T11:05:17.810 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:05:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:05:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:05:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:05:17.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:05:17.815 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:05:17.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:05:17.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:05:17.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:05:17.818 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:05:17.818 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:05:17.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:05:17.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:05:17.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:05:17.820 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:05:17.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:05:17.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:05:17.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:05:17.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:05:17.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:17.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:17.825 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:05:17.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:05:17.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:05:17.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T11:05:17.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T11:05:17.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:17.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:17.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T11:05:17.831 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T11:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T11:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T11:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T11:05:17.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T11:05:17.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-06-15T11:05:18.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T11:05:18.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:05:18.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:05:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:05:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:18.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:18.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-06-15T11:05:18.431 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T11:05:18.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T11:05:18.432 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:05:18.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:05:18.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T11:05:18.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T11:05:18.433 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:05:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:05:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:05:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:05:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:18.434 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:18.435 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:05:18.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:05:18.436 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T11:05:18.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T11:05:18.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:05:18.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:05:18.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T11:05:18.555 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T11:05:18.557 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:05:18.557 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:05:18.557 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:05:18.558 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:05:18.558 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:18.558 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:18.558 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:05:18.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:05:18.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T11:05:18.667 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T11:05:18.668 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T11:05:19.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:19.531 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:19.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:05:19.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:05:19.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-06-15T11:05:19.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:05:19.577 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:05:19.578 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3539e474-c08e-4061-87a9-adb6f23030ad 2024-06-15T11:05:19.579 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3539e474-c08e-4061-87a9-adb6f23030ad' 2024-06-15T11:05:19.580 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 3539e474-c08e-4061-87a9-adb6f23030ad 2024-06-15T11:05:19.580 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:05:19.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvdW1mvyu0IxAAOPgvGktgoE2bzWCOplEvyQ== 2024-06-15T11:05:19.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvdW1mvyu0IxAAOPgvGktgoE2bzWCOplEvyQ=="}' 2024-06-15T11:05:19.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3539e474-c08e-4061-87a9-adb6f23030ad -i td/osd-scrub-repair/0/new.json 2024-06-15T11:05:19.842 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:05:19.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T11:05:19.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBvdW1mvyu0IxAAOPgvGktgoE2bzWCOplEvyQ== --osd-uuid 3539e474-c08e-4061-87a9-adb6f23030ad 2024-06-15T11:05:19.895 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.894+0000 7f5d0453a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:19.897 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.898+0000 7f5d0453a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.898+0000 7f5d0453a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.898+0000 7f5d0453a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:05:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.898+0000 7f5d0453a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:05:19.898 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.898+0000 7f5d0453a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:05:19.977 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:19.978+0000 7f5d0453a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T11:05:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T11:05:25.885 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:05:25.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T11:05:25.888 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T11:05:25.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:05:26.145 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:05:26.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T11:05:26.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:05:26.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:05:26.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:05:26.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:05:26.179 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:26.178+0000 7f3086fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:26.207 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:26.206+0000 7f3086fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:26.209 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:26.210+0000 7f3086fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:26.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:26.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:05:26.408 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:05:26.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:26.408 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:26.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:26.758 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:26.758+0000 7f3086fde7c0 -1 Falling back to public interface 2024-06-15T11:05:27.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:27.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:27.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:05:27.656 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:05:27.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:27.657 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:27.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:28.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:28.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:28.898 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:05:28.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:05:28.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:28.899 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:29.138 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:29.380 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:29.382+0000 7f3086fde7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T11:05:30.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:30.140 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:30.140 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:05:30.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:05:30.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:30.141 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:30.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:31.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:31.479 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:05:31.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:05:31.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:31.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:31.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:32.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:32.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:32.718 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:05:32.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:05:32.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:32.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:32.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:33.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:33.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:33.984 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:05:33.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:05:33.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:33.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:34.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:35.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:35.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:35.283 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:05:35.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:05:35.283 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:35.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:35.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:36.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:36.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:36.524 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:05:36.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:05:36.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:36.525 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:36.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:37.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:37.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:37.771 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:05:37.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:05:37.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:37.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:38.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:39.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:39.014 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:05:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:05:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:39.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:39.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:40.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:40.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:40.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:05:40.321 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:05:40.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:40.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:40.565 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:41.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:41.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:41.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:05:41.568 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:05:41.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:41.568 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:42.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:43.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:43.301 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:05:43.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:05:43.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:43.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:43.651 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:44.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:44.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:44.654 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:05:44.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:05:44.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:44.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:45.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:46.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:46.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:46.236 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:05:46.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:05:46.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:46.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:46.869 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:47.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:47.876 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:47.876 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:05:47.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:05:47.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:47.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:48.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:49.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:49.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:49.245 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:05:49.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:05:49.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:49.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:49.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:50.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:50.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:50.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:05:50.684 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:05:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:50.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:51.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:52.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:05:52.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:52.070 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:05:52.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:05:52.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:52.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:05:52.309 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2875236559,v1:127.0.0.1:6803/2875236559] [v2:127.0.0.1:6804/2875236559,v1:127.0.0.1:6805/2875236559] exists,up 3539e474-c08e-4061-87a9-adb6f23030ad 2024-06-15T11:05:52.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:05:52.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:05:52.313 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:05:52.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e1dddbf-af8b-457a-b6c6-d89bd9b28220 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6e1dddbf-af8b-457a-b6c6-d89bd9b28220' 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 6e1dddbf-af8b-457a-b6c6-d89bd9b28220 2024-06-15T11:05:52.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:05:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQdW1mkM8FFBAA0aNic3bKPLKLcrg1Ne3NkQ== 2024-06-15T11:05:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQdW1mkM8FFBAA0aNic3bKPLKLcrg1Ne3NkQ=="}' 2024-06-15T11:05:52.334 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e1dddbf-af8b-457a-b6c6-d89bd9b28220 -i td/osd-scrub-repair/1/new.json 2024-06-15T11:05:52.613 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:05:52.632 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T11:05:52.633 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCQdW1mkM8FFBAA0aNic3bKPLKLcrg1Ne3NkQ== --osd-uuid 6e1dddbf-af8b-457a-b6c6-d89bd9b28220 2024-06-15T11:05:52.671 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.670+0000 7f8f6f7537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:52.698 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.698+0000 7f8f6f7537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:52.699 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.698+0000 7f8f6f7537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:52.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.702+0000 7f8f6f7537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:05:52.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.702+0000 7f8f6f7537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:05:52.700 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.702+0000 7f8f6f7537c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:05:52.775 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:52.774+0000 7f8f6f7537c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T11:05:58.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T11:05:58.848 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:05:58.850 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T11:05:58.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T11:05:58.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:05:59.121 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:05:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T11:05:59.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:05:59.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:05:59.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:05:59.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:05:59.156 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:59.158+0000 7f84eedaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:59.186 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:59.186+0000 7f84eedaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:59.187 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:59.186+0000 7f84eedaa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:05:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T11:05:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:05:59.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:05:59.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:05:59.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:05:59.734 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:05:59.734+0000 7f84eedaa7c0 -1 Falling back to public interface 2024-06-15T11:06:00.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:00.624 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:00.625 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:06:00.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:06:00.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:00.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:00.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:01.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:06:01.865 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:06:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:01.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:02.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:02.425 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:02.426+0000 7f84eedaa7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T11:06:03.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:03.107 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:03.107 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:06:03.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:06:03.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:03.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:03.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:04.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:04.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:04.542 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:06:04.542 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:06:04.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:04.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:04.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:05.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:05.911 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:05.912 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:06:05.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:06:05.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:05.912 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:06.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:07.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:07.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:07.214 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:06:07.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:06:07.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:07.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:07.455 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:08.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:08.457 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:06:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:06:08.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:08.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:08.740 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:09.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:09.742 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:09.743 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:06:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:06:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:09.743 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:10.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:11.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:11.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:11.008 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:06:11.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:06:11.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:11.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:11.344 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:12.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:12.346 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:12.346 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:06:12.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:06:12.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:12.347 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:12.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:13.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:13.587 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:13.588 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:06:13.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:06:13.588 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:13.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:13.913 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:14.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:14.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:14.915 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:06:14.915 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:06:14.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:14.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:15.604 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:16.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:16.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:16.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:06:16.606 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:06:16.606 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:16.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:17.189 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:18.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:18.191 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:18.192 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:06:18.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:06:18.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:18.192 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:18.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:19.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:19.941 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:06:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:06:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:19.942 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:20.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:21.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:21.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:21.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:06:21.889 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:06:21.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:21.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:22.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:23.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:23.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:23.430 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:06:23.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:06:23.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:23.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:24.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:25.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:25.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:25.105 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:06:25.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:06:25.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:25.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:06:25.351 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3404402181,v1:127.0.0.1:6811/3404402181] [v2:127.0.0.1:6812/3404402181,v1:127.0.0.1:6813/3404402181] exists,up 6e1dddbf-af8b-457a-b6c6-d89bd9b28220 2024-06-15T11:06:25.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:06:25.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:06:25.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:06:25.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-06-15T11:06:25.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-06-15T11:06:25.595 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' does not exist 2024-06-15T11:06:25.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-06-15T11:06:25.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-06-15T11:06:25.957 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'rbd' created 2024-06-15T11:06:25.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:06:26.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-06-15T11:06:28.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-15T11:06:28.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:06:28.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:06:28.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:06:28.627 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:06:28.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:06:28.628 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:06:28.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:06:28.628 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:06:28.629 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:06:28.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:06:28.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:06:28.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:06:28.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:06:28.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:06:28.766 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:06:29.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:06:29.002 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:06:29.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:06:29.002 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:29.003 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:06:29.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-06-15T11:06:29.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-06-15T11:06:29.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-06-15T11:06:29.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:29.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:06:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-06-15T11:06:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-06-15T11:06:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672964' 2024-06-15T11:06:29.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:29.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-06-15T11:06:29.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:29.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:06:29.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-06-15T11:06:29.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:29.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-06-15T11:06:29.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-06-15T11:06:29.313 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836492 2024-06-15T11:06:29.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:29.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-06-15T11:06:29.539 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:06:30.540 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:06:30.542 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-06-15T11:06:30.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:30.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:30.780 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-06-15T11:06:30.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:06:30.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-06-15T11:06:30.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:30.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-06-15T11:06:30.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-06-15T11:06:30.785 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672964 2024-06-15T11:06:30.785 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:31.012 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-06-15T11:06:31.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:06:31.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:31.013 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-06-15T11:06:31.261 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:06:31.262 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:06:31.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-06-15T11:06:31.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:06:31.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:31.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:31.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-06-15T11:06:31.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:06:31.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:06:31.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-06-15T11:06:31.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-06-15T11:06:32.193 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'psr_pool' created 2024-06-15T11:06:32.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:06:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-06-15T11:06:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:06:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:06:33.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:06:33.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:06:33.212 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:06:33.212 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:06:33.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:06:33.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:06:33.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:06:33.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:06:33.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:06:33.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:06:33.354 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:06:33.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:06:33.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:06:33.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:06:33.592 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:06:33.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:06:33.593 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:33.593 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:06:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-06-15T11:06:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-06-15T11:06:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-06-15T11:06:33.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:33.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:06:33.900 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-06-15T11:06:33.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-06-15T11:06:33.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-06-15T11:06:33.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:33.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-06-15T11:06:33.901 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:33.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:06:33.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-06-15T11:06:33.903 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:33.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-06-15T11:06:33.905 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836494 2024-06-15T11:06:33.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-06-15T11:06:33.905 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:34.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-06-15T11:06:34.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:34.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:34.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-06-15T11:06:34.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:06:34.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-06-15T11:06:34.177 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:34.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-06-15T11:06:34.179 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672966 2024-06-15T11:06:34.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-15T11:06:34.179 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672966 2024-06-15T11:06:34.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:06:35.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:06:35.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:35.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672966 2024-06-15T11:06:35.654 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:06:36.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T11:06:36.656 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:36.897 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-06-15T11:06:36.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:06:36.898 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:36.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:37.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:06:37.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:06:37.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:06:37.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:06:37.159 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:06:37.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:06:37.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:06:37.160 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:06:37.407 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:06:37.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:06:37.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:37.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-06-15T11:06:37.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-06-15T11:06:37.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-06-15T11:06:37.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-06-15T11:06:37.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-06-15T11:06:37.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-06-15T11:06:37.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-06-15T11:06:38.148 INFO:tasks.workunit.client.0.smithi116.stderr:noscrub is unset 2024-06-15T11:06:38.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-06-15T11:06:38.575 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is unset 2024-06-15T11:06:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-06-15T11:06:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-06-15T11:06:38.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-06-15T11:06:38.840 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-06-15T11:06:38.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-06-15T11:06:38.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-06-15T11:06:38.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-06-15T11:06:38.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T11:06:39.087 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-06-15T11:06:39.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-06-15T11:06:39.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-06-15T11:06:39.088 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-06-15T11:06:39.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-06-15T11:06:39.089 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T11:06:39.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-06-15T11:06:39.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:06:39.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:06:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:06:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T11:06:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:06:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:06:39.683 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:06:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:06:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:06:39.684 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:06:41.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:06:41.676 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:06:41.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:06:41.678 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:06:41.678 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:06:41.679 INFO:tasks.workunit.client.0.smithi116.stderr:start osd.0 2024-06-15T11:06:41.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-06-15T11:06:41.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:06:41.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:06:41.681 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:06:41.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:06:41.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:06:41.715 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:41.713+0000 7f0b4c9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:06:41.718 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:41.717+0000 7f0b4c9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:06:41.719 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:41.717+0000 7f0b4c9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:0 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:41.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:06:42.166 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:42.278 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:42.277+0000 7f0b4c9ff7c0 -1 Falling back to public interface 2024-06-15T11:06:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:06:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:06:43.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:43.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:06:43.406 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:06:44.409 INFO:tasks.workunit.client.0.smithi116.stderr:2 2024-06-15T11:06:44.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:44.410 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:06:44.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:06:44.920 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:06:44.921+0000 7f0b4c9ff7c0 -1 osd.0 24 log_to_monitors true 2024-06-15T11:06:45.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:06:45.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:06:45.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:06:45.652 INFO:tasks.workunit.client.0.smithi116.stderr:3 2024-06-15T11:06:45.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:06:45.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:06:46.027 INFO:tasks.workunit.client.0.smithi116.stderr:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/3508348642,v1:127.0.0.1:6803/3508348642] [v2:127.0.0.1:6804/3508348642,v1:127.0.0.1:6805/3508348642] exists,up 3539e474-c08e-4061-87a9-adb6f23030ad 2024-06-15T11:06:46.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:06:46.029 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:06:46.030 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:06:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:06:46.031 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:06:46.031 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:06:46.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:06:46.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:06:46.153 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:06:46.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:06:46.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:06:46.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:06:46.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:06:46.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:06:46.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:06:46.427 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:06:46.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:06:46.428 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:46.428 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:06:46.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084290 2024-06-15T11:06:46.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084290 2024-06-15T11:06:46.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290' 2024-06-15T11:06:46.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:46.622 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:06:46.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-06-15T11:06:46.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-06-15T11:06:46.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-42949672970' 2024-06-15T11:06:46.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:46.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084290 2024-06-15T11:06:46.771 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:46.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:06:46.772 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084290 2024-06-15T11:06:46.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084290 2024-06-15T11:06:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2024-06-15T11:06:46.774 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.0 seq 120259084290 2024-06-15T11:06:46.774 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084290 2024-06-15T11:06:47.004 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:06:48.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:06:48.007 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:48.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084290 -lt 120259084290 2024-06-15T11:06:48.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:48.250 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-06-15T11:06:48.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:48.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:06:48.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-06-15T11:06:48.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:48.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-06-15T11:06:48.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-06-15T11:06:48.256 INFO:tasks.workunit.client.0.smithi116.stderr:waiting osd.1 seq 42949672970 2024-06-15T11:06:48.256 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:48.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-06-15T11:06:48.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:06:48.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:48.494 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:48.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-06-15T11:06:48.772 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:06:48.773 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:06:49.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-06-15T11:06:49.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:06:49.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:06:49.027 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:06:49.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-06-15T11:06:49.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:06:49.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:06:49.287 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-06-15T11:06:49.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-06-15T11:06:49.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-15T11:06:49.317 INFO:tasks.workunit.client.0.smithi116.stderr:No scrub information available for pg 2.0 2024-06-15T11:06:49.317 INFO:tasks.workunit.client.0.smithi116.stderr:error 2: (2) No such file or directory 2024-06-15T11:06:49.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-06-15T11:06:49.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-06-15T11:06:49.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-06-15T11:06:49.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-15T11:06:49.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:49.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T11:06:49.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:49.322 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-15T11:06:49.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-06-15T11:06:32.146573+0000 2024-06-15T11:06:49.541 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-06-15T11:06:49.765 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-06-15T11:06:49.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-06-15T11:06:32.146573+0000 last_deep_scrub_stamp 2024-06-15T11:06:49.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T11:06:49.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:06:32.146573+0000 2024-06-15T11:06:49.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-06-15T11:06:49.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:06:49.783 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:49.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-15T11:06:49.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:49.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T11:06:49.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:49.784 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-15T11:06:50.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:32.146573+0000 '>' 2024-06-15T11:06:32.146573+0000 2024-06-15T11:06:50.014 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:06:51.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:06:51.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:51.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-15T11:06:51.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:51.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T11:06:51.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:51.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-15T11:06:51.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:32.146573+0000 '>' 2024-06-15T11:06:32.146573+0000 2024-06-15T11:06:51.251 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:06:52.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:06:52.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:52.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-06-15T11:06:52.254 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:52.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-06-15T11:06:52.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:52.255 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-06-15T11:06:52.492 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:32.146573+0000 2024-06-15T11:06:52.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:06:52.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-15T11:06:52.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-06-15T11:06:52.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-15T11:06:52.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-15T11:06:52.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:06:52.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:06:52.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:06:52.753 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:06:52.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:06:52.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:52.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:06:52.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084292 2024-06-15T11:06:52.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084292 2024-06-15T11:06:52.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084292' 2024-06-15T11:06:52.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:06:52.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:06:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-06-15T11:06:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-06-15T11:06:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084292 1-42949672972' 2024-06-15T11:06:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:53.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084292 2024-06-15T11:06:53.063 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:53.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:06:53.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084292 2024-06-15T11:06:53.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:53.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084292 2024-06-15T11:06:53.066 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2024-06-15T11:06:53.066 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 120259084292 2024-06-15T11:06:53.066 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:53.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084291 -lt 120259084292 2024-06-15T11:06:53.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:06:54.290 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:06:54.291 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:06:54.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084292 -lt 120259084292 2024-06-15T11:06:54.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:06:54.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-06-15T11:06:54.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:06:54.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:06:54.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-06-15T11:06:54.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:06:54.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-06-15T11:06:54.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-06-15T11:06:54.531 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672972 2024-06-15T11:06:54.532 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:06:54.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-06-15T11:06:54.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-06-15T11:06:54.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:54.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:06:54.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:54.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:06:55.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:55.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-15T11:06:55.200 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:06:55.200 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": false, 2024-06-15T11:06:55.201 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T11:06:55.201 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-08T11:05:15.203699+0000" 2024-06-15T11:06:55.201 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:06:55.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:55.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T11:06:55.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:55.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:06:55.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:06:55.215 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:55.215 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:06:55.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:55.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:06:55.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:55.216 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:06:55.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:55.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:06:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:06:56.443 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:56.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:06:56.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:56.444 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:06:56.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:56.445 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:06:56.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:56.685 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:06:57.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:06:57.686 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:57.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:06:57.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:57.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:06:57.688 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:57.689 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:06:57.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:57.929 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:06:58.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:06:58.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:06:58.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:06:58.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:06:58.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:06:58.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:06:58.933 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:06:59.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:06:59.163 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:00.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:00.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:00.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:00.165 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:00.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:00.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:00.166 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:00.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:07:00.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:01.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:01.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:01.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:01.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:01.405 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:01.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:01.406 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:01.644 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:49.976174+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:07:01.645 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:02.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:02.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:02.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:02.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:02.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:02.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:02.648 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:02.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:06:55.983967+0000 '>' 2024-06-15T11:06:49.976174+0000 2024-06-15T11:07:02.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:07:02.887 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-06-15T11:07:02.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-15T11:07:02.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-06-15T11:07:02.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-15T11:07:02.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-06-15T11:07:03.240 INFO:tasks.workunit.client.0.smithi116.stderr:nodeep-scrub is set 2024-06-15T11:07:03.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-06-15T11:07:03.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-15T11:07:03.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:07:03.426 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:07:03.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:07:03.675 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:07:03.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:07:03.675 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:07:03.676 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:07:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084295 2024-06-15T11:07:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084295 2024-06-15T11:07:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295' 2024-06-15T11:07:03.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:07:03.819 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295 1-42949672975' 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084295 2024-06-15T11:07:03.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:07:03.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:07:03.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084295 2024-06-15T11:07:03.964 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:07:03.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084295 2024-06-15T11:07:03.966 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 120259084295 2024-06-15T11:07:03.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-06-15T11:07:03.966 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:07:04.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084294 -lt 120259084295 2024-06-15T11:07:04.175 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:07:05.177 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:07:05.178 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:07:05.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084294 -lt 120259084295 2024-06-15T11:07:05.411 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:07:06.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T11:07:06.414 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:07:06.655 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084296 -lt 120259084295 2024-06-15T11:07:06.656 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:07:06.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:07:06.657 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-06-15T11:07:06.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:07:06.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-06-15T11:07:06.660 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:07:06.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-06-15T11:07:06.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-06-15T11:07:06.662 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672975 2024-06-15T11:07:06.662 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:07:06.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-06-15T11:07:06.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-06-15T11:07:11.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-06-15T11:07:12.014 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:07:12.014 INFO:tasks.workunit.client.0.smithi116.stdout: "deep": false, 2024-06-15T11:07:12.014 INFO:tasks.workunit.client.0.smithi116.stdout: "must": false, 2024-06-15T11:07:12.014 INFO:tasks.workunit.client.0.smithi116.stdout: "stamp": "2024-06-08T11:05:32.017127+0000" 2024-06-15T11:07:12.014 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:07:12.028 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-06-15T11:07:12.029 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-06-15T11:07:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-06-15T11:07:12.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-06-15T11:07:13.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-06-15T11:07:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-06-15T11:07:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-06-15T11:07:13.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-06-15T11:07:13.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-15T11:07:13.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-06-15T11:07:13.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-06-15T11:07:13.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-06-15T11:07:13.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:07:13.081 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:07:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:07:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:07:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:07:13.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:07:13.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:07:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-06-15T11:07:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-06-15T11:07:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298' 2024-06-15T11:07:13.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:07:13.466 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:07:13.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-06-15T11:07:13.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-06-15T11:07:13.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298 1-42949672978' 2024-06-15T11:07:13.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:07:13.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084298 2024-06-15T11:07:13.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:07:13.596 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:07:13.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084298 2024-06-15T11:07:13.597 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:07:13.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-06-15T11:07:13.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084298' 2024-06-15T11:07:13.599 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 120259084298 2024-06-15T11:07:13.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:07:13.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084297 -lt 120259084298 2024-06-15T11:07:13.814 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:07:14.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:07:14.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:07:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084298 -lt 120259084298 2024-06-15T11:07:15.047 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:07:15.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-06-15T11:07:15.048 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:07:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:07:15.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-06-15T11:07:15.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:07:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-06-15T11:07:15.052 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672978 2024-06-15T11:07:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-06-15T11:07:15.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:07:15.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-06-15T11:07:15.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-06-15T11:07:15.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-06-15T11:07:15.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-06-15T11:07:15.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:15.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:15.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:15.282 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:15.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:15.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-06-15T11:07:15.719 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 2.0 on osd.0 to scrub 2024-06-15T11:07:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-06-15T11:07:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:07:15.734 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:07:15.735 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:15.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:15.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:15.735 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:15.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:15.736 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:15.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-08T11:05:32.017127+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:15.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:16.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:16.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:16.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:16.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:16.962 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:16.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:16.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:17.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-08T11:05:32.017127+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:17.202 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:18.203 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:18.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:18.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:18.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:18.204 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:18.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:18.205 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:18.445 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-08T11:05:32.017127+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:18.446 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:19.450 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:19.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-08T11:05:32.017127+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:19.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:20.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:20.691 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:20.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:20.692 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:20.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:20.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:20.693 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:20.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-08T11:05:32.017127+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:20.937 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:07:21.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:07:21.939 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:07:21.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-06-15T11:07:21.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-06-15T11:07:21.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:07:21.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-06-15T11:07:21.941 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:07:22.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:07:15.904243+0000 '>' 2024-06-08T11:05:32.017127+0000 2024-06-15T11:07:22.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:07:22.176 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-06-15T11:07:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-06-15T11:07:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-06-15T11:07:22.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-06-15T11:07:22.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T11:07:22.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:07:22.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:07:22.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:07:22.212 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:07:22.213 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:07:22.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:07:22.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:07:22.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:07:22.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:07:22.523 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:07:22.524 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:07:22.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:07:22.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:07:22.527 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:07:22.528 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:07:22.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:07:22.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:07:22.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:07:22.530 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:07:22.531 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:07:22.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:07:22.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:07:22.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:07:22.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:22.550 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:22.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:07:22.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:07:22.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:07:22.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:07:22.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:07:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:07:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:07:22.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:07:22.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:07:22.556 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:07:22.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:07:22.558 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:07:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:07:22.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:07:22.559 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:07:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:07:22.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:07:22.561 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:07:22.561 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:07:22.562 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:07:22.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:07:22.564 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:07:22.565 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:07:22.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:22.566 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:22.566 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:07:22.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:07:22.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:07:22.567 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T11:07:22.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T11:07:22.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:22.569 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:22.569 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T11:07:22.570 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T11:07:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T11:07:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T11:07:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T11:07:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-06-15T11:07:22.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T11:07:22.573 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T11:07:22.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T11:07:22.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:07:22.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:07:22.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:07:22.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:22.955 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:22.956 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:22.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T11:07:23.180 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T11:07:23.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T11:07:23.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:07:23.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:07:23.182 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:23.183 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:23.184 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:07:23.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:07:23.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T11:07:23.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T11:07:23.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:07:23.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:07:23.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T11:07:23.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T11:07:23.301 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:23.302 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:07:23.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:07:23.303 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T11:07:23.412 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:24.266 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:24.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:24.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:07:24.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:07:24.303 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:07:24.304 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-06-15T11:07:24.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-06-15T11:07:24.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:07:24.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:24.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:07:24.311 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87410f34-4ed2-48f1-a367-010332adf9f0 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 87410f34-4ed2-48f1-a367-010332adf9f0 2024-06-15T11:07:24.312 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 87410f34-4ed2-48f1-a367-010332adf9f0' 2024-06-15T11:07:24.313 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:07:24.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsdW1mBiK/ExAA+Ml7rZ05LOXsm8f57YZlNw== 2024-06-15T11:07:24.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsdW1mBiK/ExAA+Ml7rZ05LOXsm8f57YZlNw=="}' 2024-06-15T11:07:24.329 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87410f34-4ed2-48f1-a367-010332adf9f0 -i td/osd-scrub-repair/0/new.json 2024-06-15T11:07:24.568 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:07:24.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T11:07:24.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDsdW1mBiK/ExAA+Ml7rZ05LOXsm8f57YZlNw== --osd-uuid 87410f34-4ed2-48f1-a367-010332adf9f0 2024-06-15T11:07:24.616 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.616+0000 7f8e0a4fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:24.619 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.616+0000 7f8e0a4fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:24.620 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.620+0000 7f8e0a4fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:24.620 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.620+0000 7f8e0a4fd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:07:24.620 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.620+0000 7f8e0a4fd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:07:24.620 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.620+0000 7f8e0a4fd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:07:24.720 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:24.716+0000 7f8e0a4fd7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T11:07:30.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T11:07:30.379 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:07:30.380 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T11:07:30.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T11:07:30.381 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:07:30.652 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:07:30.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T11:07:30.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:07:30.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:07:30.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:07:30.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:07:30.684 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:30.684+0000 7fdc153097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:30.687 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:30.684+0000 7fdc153097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:30.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:30.688+0000 7fdc153097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:30.893 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:30.894 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:07:30.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:07:30.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:30.895 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:31.123 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:31.223 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:31.220+0000 7fdc153097c0 -1 Falling back to public interface 2024-06-15T11:07:32.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:32.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:32.126 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:07:32.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:07:32.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:32.126 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:32.369 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:33.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:33.371 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:33.372 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:07:33.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:07:33.372 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:33.373 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:33.614 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:33.736 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:33.732+0000 7fdc153097c0 -1 osd.0 0 log_to_monitors true 2024-06-15T11:07:34.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:34.616 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:34.616 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:07:34.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:07:34.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:34.617 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:34.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:35.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:35.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:35.905 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:07:35.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:07:35.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:35.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:36.148 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:37.150 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:37.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:37.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:07:37.151 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:07:37.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:37.152 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:37.415 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:38.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:38.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:38.417 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:07:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:07:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:38.418 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:38.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:39.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:39.717 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:39.717 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:07:39.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:07:39.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:39.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:39.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:40.958 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:07:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:07:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:40.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:41.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:42.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:42.198 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:42.198 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:07:42.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:07:42.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:42.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:42.440 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:43.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:43.441 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:43.442 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:07:43.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:07:43.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:43.442 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:43.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:44.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:44.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:44.755 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:07:44.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:07:44.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:44.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:45.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:45.999 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:45.999 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:07:46.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:07:46.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:46.000 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:46.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:47.731 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:07:47.732 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:07:47.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:47.732 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:48.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:49.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:49.284 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:49.284 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:07:49.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:07:49.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:49.285 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:50.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:51.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:51.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:51.009 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:07:51.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:07:51.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:51.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:51.457 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:52.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:52.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:07:52.460 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:07:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:52.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:53.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:54.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:54.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:54.080 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:07:54.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:07:54.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:54.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:54.324 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:55.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:55.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:55.326 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:07:55.326 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:07:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:55.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:55.650 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:07:56.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:07:56.652 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:07:56.652 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:07:56.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:07:56.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:07:56.653 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:07:56.913 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3365899437,v1:127.0.0.1:6803/3365899437] [v2:127.0.0.1:6804/3365899437,v1:127.0.0.1:6805/3365899437] exists,up 87410f34-4ed2-48f1-a367-010332adf9f0 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:07:56.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:07:56.917 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:07:56.918 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:07:56.918 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:07:56.918 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:07:56.918 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:07:56.919 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:07:56.920 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:07:56.921 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:07:56.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9c15705-12a3-4611-b796-4dd0e0de59d3 2024-06-15T11:07:56.921 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e9c15705-12a3-4611-b796-4dd0e0de59d3' 2024-06-15T11:07:56.921 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 e9c15705-12a3-4611-b796-4dd0e0de59d3 2024-06-15T11:07:56.922 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:07:56.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMdm1mKMddOBAAg3QG8nnKphlHujg08W7xFA== 2024-06-15T11:07:56.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMdm1mKMddOBAAg3QG8nnKphlHujg08W7xFA=="}' 2024-06-15T11:07:56.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9c15705-12a3-4611-b796-4dd0e0de59d3 -i td/osd-scrub-repair/1/new.json 2024-06-15T11:07:57.225 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:07:57.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T11:07:57.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAMdm1mKMddOBAAg3QG8nnKphlHujg08W7xFA== --osd-uuid e9c15705-12a3-4611-b796-4dd0e0de59d3 2024-06-15T11:07:57.278 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.276+0000 7fb7875997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:57.292 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.292+0000 7fb7875997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:57.293 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.292+0000 7fb7875997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:07:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.292+0000 7fb7875997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:07:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.292+0000 7fb7875997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:07:57.294 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.292+0000 7fb7875997c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:07:57.373 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:07:57.372+0000 7fb7875997c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T11:08:03.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T11:08:03.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:08:03.186 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T11:08:03.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T11:08:03.187 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:08:03.465 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:08:03.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T11:08:03.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:08:03.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:08:03.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:08:03.467 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:08:03.499 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:08:03.496+0000 7fded92cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:08:03.501 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:08:03.500+0000 7fded92cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:08:03.503 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:08:03.500+0000 7fded92cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:08:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T11:08:03.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:08:03.759 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:03.760 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:04.013 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:04.057 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:08:04.056+0000 7fded92cb7c0 -1 Falling back to public interface 2024-06-15T11:08:05.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:05.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:05.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:08:05.015 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:08:05.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:05.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:05.255 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:06.256 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:06.257 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:06.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:06.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:06.586 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:08:06.584+0000 7fded92cb7c0 -1 osd.1 0 log_to_monitors true 2024-06-15T11:08:07.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:07.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:07.549 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:08:07.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:08:07.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:07.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:07.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:08.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:08.795 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:08.795 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:08:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:08:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:08.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:09.032 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:10.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:10.034 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:10.035 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:08:10.035 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:08:10.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:10.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:10.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:11.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:11.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:11.278 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:08:11.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:08:11.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:11.278 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:11.528 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:12.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:12.529 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:12.530 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:08:12.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:08:12.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:12.530 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:12.771 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:13.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:13.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:13.773 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:08:13.774 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:08:13.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:13.774 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:14.016 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:15.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:15.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:15.019 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:08:15.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:08:15.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:15.020 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:15.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:16.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:16.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:16.259 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:08:16.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:08:16.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:16.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:16.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:17.611 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:08:17.612 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:08:17.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:17.612 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:17.916 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:18.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:18.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:18.918 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:08:18.919 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:08:18.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:18.919 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:19.161 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:20.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:20.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:20.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:08:20.164 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:08:20.164 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:20.165 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:20.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:21.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:21.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:21.768 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:08:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:08:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:21.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:22.554 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:23.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:23.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:23.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:08:23.557 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:08:23.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:23.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:24.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:25.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:25.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:25.026 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:08:25.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:08:25.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:25.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:25.463 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:26.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:26.465 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:26.465 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:08:26.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:08:26.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:26.466 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:26.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:27.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:27.966 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:08:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:08:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:27.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:28.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:29.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:29.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:29.558 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:08:29.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:08:29.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:29.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:29.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:08:30.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:08:30.906 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:08:30.906 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T11:08:30.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T11:08:30.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:08:30.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:08:31.145 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/376018555,v1:127.0.0.1:6811/376018555] [v2:127.0.0.1:6812/376018555,v1:127.0.0.1:6813/376018555] exists,up e9c15705-12a3-4611-b796-4dd0e0de59d3 2024-06-15T11:08:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:08:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:08:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:08:31.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-06-15T11:08:31.147 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-06-15T11:08:31.489 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T11:08:31.511 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:08:32.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-06-15T11:08:33.001 INFO:tasks.workunit.client.0.smithi116.stderr:set pool 1 size to 2 2024-06-15T11:08:33.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-06-15T11:08:33.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:08:33.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:08:33.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:08:33.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:08:33.025 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:08:33.025 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:08:33.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:08:33.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:08:33.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:08:33.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:08:33.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:08:33.473 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:08:33.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:08:33.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:08:33.473 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:08:33.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-06-15T11:08:33.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-06-15T11:08:33.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-06-15T11:08:33.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:08:33.680 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:08:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-06-15T11:08:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-06-15T11:08:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672966' 2024-06-15T11:08:33.941 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:08:33.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-06-15T11:08:33.942 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:08:33.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:08:33.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-06-15T11:08:33.945 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:08:33.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-06-15T11:08:33.947 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836497 2024-06-15T11:08:33.947 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-06-15T11:08:33.947 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:08:34.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-06-15T11:08:34.265 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:08:35.267 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:08:35.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:08:35.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-06-15T11:08:35.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:08:35.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-06-15T11:08:35.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:08:35.519 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:08:35.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-06-15T11:08:35.520 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:08:35.521 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-06-15T11:08:35.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-06-15T11:08:35.522 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672966 2024-06-15T11:08:35.522 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:08:35.762 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-06-15T11:08:35.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:08:35.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:08:35.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:08:36.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:08:36.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:08:36.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:08:36.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-06-15T11:08:36.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:08:36.300 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:08:36.301 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-06-15T11:08:36.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-06-15T11:08:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-06-15T11:08:36.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:08:36.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:08:36.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:08:36.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:08:36.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:08:36.705 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:08:36.706 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:08:36.938 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:08:36.939 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:08:36.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:08:36.940 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:08:37.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:08:37.194 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:08:37.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:08:37.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-06-15T11:08:37.195 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-06-15T11:08:37.195 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-06-15T11:08:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.196 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:37.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:37.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:37.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:37.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.836 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:37.986 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:37.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.413 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.706 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:38.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:38.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.677 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:39.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.388 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.389 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.689 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:40.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:41.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:41.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:41.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:41.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:41.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-06-15T11:08:41.375 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:08:41.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-06-15T11:08:41.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T11:08:41.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-15T11:08:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-15T11:08:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T11:08:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T11:08:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T11:08:41.520 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T11:08:41.763 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T11:08:41.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-15T11:08:41.764 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T11:08:42.024 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-06-15T11:08:42.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-06-15T11:08:42.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T11:08:42.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-15T11:08:42.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T11:08:42.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-15T11:08:42.282 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-06-15T11:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-06-15T11:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T11:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T11:08:42.283 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T11:08:42.284 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T11:08:42.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-06-15T11:08:42.552 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T11:08:42.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:08:42.553 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:08:42.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:08:42.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:08:42.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:08:42.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:08:42.863 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T11:08:42.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:08:42.864 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:08:42.864 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:08:42.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:08:42.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:08:44.179 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:08:44.181 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-06-15T11:08:44.183 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:44.184 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-06-15T11:08:44.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:44.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-15T11:08:44.185 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:44.186 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-15T11:08:45.300 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:ff7b1f36:::obj1:head# 2024-06-15T11:08:45.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:45.835 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-06-15T11:08:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-15T11:08:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:45.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:45.838 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-15T11:08:46.953 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:104778fc:::obj2:head# 2024-06-15T11:08:47.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:47.499 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-06-15T11:08:47.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:47.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-15T11:08:47.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:47.502 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-15T11:08:48.316 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:8dd16f86:::obj3:head# 2024-06-15T11:08:48.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:48.860 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:48.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:48.862 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-15T11:08:49.811 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:0ee9ae15:::obj4:head# 2024-06-15T11:08:50.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:50.385 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-15T11:08:51.062 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:c52c9666:::obj5:head# 2024-06-15T11:08:51.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:51.599 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-06-15T11:08:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-15T11:08:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:51.601 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:51.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:51.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:51.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-15T11:08:52.282 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:233a42c1:::obj6:head# 2024-06-15T11:08:52.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:52.817 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-06-15T11:08:52.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:52.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-15T11:08:52.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:52.819 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:52.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:52.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:52.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:52.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-15T11:08:53.559 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:9a09113a:::obj7:head# 2024-06-15T11:08:54.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:54.095 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-06-15T11:08:54.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:54.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:54.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-15T11:08:54.865 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:a557efb1:::obj8:head# 2024-06-15T11:08:55.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:55.403 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-06-15T11:08:55.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:55.404 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:55.405 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-15T11:08:56.128 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:fe351e27:::obj9:head# 2024-06-15T11:08:56.666 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:56.667 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:56.669 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-15T11:08:57.314 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:d1337354:::obj10:head# 2024-06-15T11:08:57.850 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:57.851 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-06-15T11:08:57.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:08:57.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-15T11:08:57.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:57.853 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:08:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:08:57.854 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-15T11:08:58.607 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:eeae2a94:::obj11:head# 2024-06-15T11:08:59.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:08:59.394 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:08:59.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-15T11:09:00.220 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:af5f95cb:::obj12:head# 2024-06-15T11:09:01.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:01.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-06-15T11:09:01.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:09:01.010 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:09:01.011 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-15T11:09:01.757 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:a61ad63e:::obj13:head# 2024-06-15T11:09:02.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:02.299 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:02.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-15T11:09:02.974 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:e0a44829:::obj14:head# 2024-06-15T11:09:03.510 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:03.511 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:09:03.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-15T11:09:04.126 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:ab946124:::obj15:head# 2024-06-15T11:09:04.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:04.663 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:04.665 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-15T11:09:05.328 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:461f8b5e:::obj16:head# 2024-06-15T11:09:05.861 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:05.863 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-06-15T11:09:05.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:09:05.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-15T11:09:05.865 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:05.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:05.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:09:05.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:05.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:09:05.866 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-15T11:09:06.604 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:24fc6e92:::obj17:head# 2024-06-15T11:09:07.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:07.155 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:07.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:07.158 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-15T11:09:07.806 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:b9836a99:::obj18:head# 2024-06-15T11:09:08.349 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:08.350 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-06-15T11:09:08.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-06-15T11:09:08.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-15T11:09:08.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:09:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:09:08.352 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-15T11:09:09.050 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:00b31b6c:::obj19:head# 2024-06-15T11:09:09.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-06-15T11:09:09.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-06-15T11:09:09.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-06-15T11:09:09.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-15T11:09:09.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:09:09.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:09:09.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:09:09.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:09:09.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:09.598 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-15T11:09:10.285 INFO:tasks.workunit.client.0.smithi116.stdout:remove #1:24b5611b:::obj20:head# 2024-06-15T11:09:10.822 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:09:10.823 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:09:10.824 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:09:10.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:09:10.824 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:09:10.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:09:10.825 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:09:10.825 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:10.825 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:10.825 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:10.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:10.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:09:10.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:09:10.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:09:10.826 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T11:09:10.827 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:09:10.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:10.829 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T11:09:10.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T11:09:10.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:09:10.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:09:10.834 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:09:10.863 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:10.863+0000 7f5a745c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:10.866 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:10.867+0000 7f5a745c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:10.867 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:10.867+0000 7f5a745c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:11.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T11:09:11.070 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:09:11.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:09:11.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:11.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:09:11.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:11.415 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:11.415+0000 7f5a745c77c0 -1 Falling back to public interface 2024-06-15T11:09:12.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:12.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:12.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:09:12.317 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:09:12.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:12.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:09:12.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:13.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:13.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:13.559 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:09:13.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:09:13.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:13.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:09:13.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:13.939 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:13.939+0000 7f5a745c77c0 -1 osd.1 18 log_to_monitors true 2024-06-15T11:09:14.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:14.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:14.805 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:09:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:09:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:14.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:09:15.046 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/3256531016,v1:127.0.0.1:6803/3256531016] [v2:127.0.0.1:6804/3256531016,v1:127.0.0.1:6805/3256531016] exists,up e9c15705-12a3-4611-b796-4dd0e0de59d3 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:09:15.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:09:15.049 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:09:15.050 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:15.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:09:15.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:09:15.054 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:09:15.054 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:15.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:09:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:09:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:09:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:09:15.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:09:15.080 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:15.083+0000 7f65a7c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:15.108 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:15.111+0000 7f65a7c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:15.109 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:15.111+0000 7f65a7c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:09:15.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:09:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:09:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:09:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:09:15.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:15.298 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:09:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:09:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:15.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:15.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:15.650 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:15.651+0000 7f65a7c727c0 -1 Falling back to public interface 2024-06-15T11:09:16.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:16.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:16.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:09:16.550 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:09:16.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:16.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:16.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:17.788 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:17.788 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:09:17.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:09:17.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:17.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:18.130 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:18.131+0000 7f65a7c727c0 -1 osd.0 16 log_to_monitors true 2024-06-15T11:09:19.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:19.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:19.026 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:09:19.027 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:09:19.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:19.027 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:19.285 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/1212344766,v1:127.0.0.1:6811/1212344766] [v2:127.0.0.1:6812/1212344766,v1:127.0.0.1:6813/1212344766] exists,up 87410f34-4ed2-48f1-a367-010332adf9f0 2024-06-15T11:09:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:09:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:09:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:09:19.288 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:09:19.289 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:09:19.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:09:19.290 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:09:19.401 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:09:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:09:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:09:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:09:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:19.646 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:09:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-06-15T11:09:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-06-15T11:09:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-06-15T11:09:19.793 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:19.793 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:09:19.931 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313219 2024-06-15T11:09:19.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313219 2024-06-15T11:09:19.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-90194313219' 2024-06-15T11:09:19.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:19.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-06-15T11:09:19.932 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:19.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:09:19.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-06-15T11:09:19.934 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:19.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-06-15T11:09:19.935 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-06-15T11:09:19.936 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 103079215106 2024-06-15T11:09:19.936 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:20.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-06-15T11:09:20.155 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:09:21.156 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:09:21.157 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:21.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-06-15T11:09:21.419 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:21.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313219 2024-06-15T11:09:21.420 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:21.421 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:09:21.422 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313219 2024-06-15T11:09:21.423 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:21.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313219 2024-06-15T11:09:21.424 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 90194313219 2024-06-15T11:09:21.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313219' 2024-06-15T11:09:21.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:09:21.674 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313219 -lt 90194313219 2024-06-15T11:09:21.674 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:09:21.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:09:21.675 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:09:21.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:09:21.924 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:09:21.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:09:21.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:09:21.925 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:09:21.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:09:21.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:09:21.926 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:09:22.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:09:22.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:09:22.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:09:22.175 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:09:22.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:09:22.430 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:09:22.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:09:22.431 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:09:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:22.673 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-15T11:09:22.886 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.0 on osd.1 to repair 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:09:22.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:09:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T11:09:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:09:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:09:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:09:22.904 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:09:23.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:08:31.442914+0000 '>' 2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:23.130 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:09:24.131 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:09:24.132 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:09:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T11:09:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:09:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:09:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:09:24.133 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:09:24.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:08:31.442914+0000 '>' 2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:24.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:09:25.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:09:25.368 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:09:25.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T11:09:25.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:09:25.369 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:09:25.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:09:25.370 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:09:23.066389+0000 '>' 2024-06-15T11:08:31.442914+0000 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:09:25.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:09:25.610 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:09:25.611 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:09:25.611 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:09:25.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:09:25.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:09:25.611 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:09:25.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:09:25.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:09:25.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:09:25.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:09:25.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:09:25.724 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:09:25.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:09:25.958 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:09:25.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:09:25.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:25.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:09:26.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215108 2024-06-15T11:09:26.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215108 2024-06-15T11:09:26.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108' 2024-06-15T11:09:26.129 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:26.129 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:09:26.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313221 2024-06-15T11:09:26.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313221 2024-06-15T11:09:26.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-90194313221' 2024-06-15T11:09:26.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:26.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215108 2024-06-15T11:09:26.270 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:26.271 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:09:26.272 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215108 2024-06-15T11:09:26.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:26.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215108 2024-06-15T11:09:26.274 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 103079215108 2024-06-15T11:09:26.274 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-06-15T11:09:26.274 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:26.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215107 -lt 103079215108 2024-06-15T11:09:26.532 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:09:27.533 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:09:27.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:27.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215108 2024-06-15T11:09:27.776 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:27.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313221 2024-06-15T11:09:27.777 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:27.779 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:09:27.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313221 2024-06-15T11:09:27.781 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:27.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313221 2024-06-15T11:09:27.782 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 90194313221 2024-06-15T11:09:27.782 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313221' 2024-06-15T11:09:27.783 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:09:28.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313221 -lt 90194313221 2024-06-15T11:09:28.017 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:09:28.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:09:28.018 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:09:28.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:09:28.286 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:09:28.287 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:09:28.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:09:28.539 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:09:28.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:09:28.540 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:09:28.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:09:28.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:09:28.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:09:28.792 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-06-15T11:09:29.015 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T11:09:29.016 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-15T11:09:23.812138+0000 16'30 25:124 [1,0] 1 [1,0] 1 16'30 2024-06-15T11:09:23.066389+0000 16'30 2024-06-15T11:09:23.066389+0000 0 1 periodic scrub scheduled @ 2024-06-16T11:09:23.066389+0000 20 0 2024-06-15T11:09:29.016 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T11:09:29.016 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T11:09:29.016 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T11:09:29.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-06-15T11:09:29.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:09:29.032 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:09:29.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:09:29.263 INFO:tasks.workunit.client.0.smithi116.stderr:1' 2024-06-15T11:09:29.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:09:29.263 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:29.264 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:09:29.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-06-15T11:09:29.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-06-15T11:09:29.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-06-15T11:09:29.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:09:29.425 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:09:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313223 2024-06-15T11:09:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313223 2024-06-15T11:09:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-90194313223' 2024-06-15T11:09:29.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:29.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:29.585 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-06-15T11:09:29.586 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:09:29.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-06-15T11:09:29.587 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:29.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-06-15T11:09:29.589 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 103079215110 2024-06-15T11:09:29.589 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-06-15T11:09:29.589 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:29.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-06-15T11:09:29.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:09:30.813 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:09:30.814 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:09:31.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215110 -lt 103079215110 2024-06-15T11:09:31.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:09:31.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-90194313223 2024-06-15T11:09:31.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:09:31.053 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:09:31.054 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-90194313223 2024-06-15T11:09:31.055 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:09:31.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313223 2024-06-15T11:09:31.056 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 90194313223' 2024-06-15T11:09:31.056 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 90194313223 2024-06-15T11:09:31.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:09:31.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313223 -lt 90194313223 2024-06-15T11:09:31.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-06-15T11:09:31.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 210, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 30, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 60, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2024-06-15T11:09:31.488 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 30, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 30, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 30, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 20, 2024-06-15T11:09:31.489 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 140, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2024-06-15T11:09:31.490 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 20 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:09:31.491 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-06-15T11:09:31.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-06-15T11:09:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-06-15T11:09:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-06-15T11:09:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:31.626 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-06-15T11:09:31.818 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:31.832 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:09:31.832 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 1, 2024-06-15T11:09:31.832 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 21, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 90194313223, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 1, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 1, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 1, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 1, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 104857600, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 28560, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 328, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 4, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 28219, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 104829040, 2024-06-15T11:09:31.833 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 107374182400, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 107344936960, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 335872, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 55896, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2024-06-15T11:09:31.834 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 4398, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 28896978 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [ 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: 0 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 10, 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2024-06-15T11:09:31.836 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [] 2024-06-15T11:09:31.837 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:09:31.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:31.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-06-15T11:09:32.044 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:32.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-06-15T11:09:32.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-06-15T11:09:32.062 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-06-15T11:09:32.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:32.063 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-15T11:09:32.262 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 24, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 103079215110, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 1, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 1, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 1, 2024-06-15T11:09:32.277 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 1, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 104857600, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 28560, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 328, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 4, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 28219, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 104829040, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 107374182400, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 107344936960, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 335872, 2024-06-15T11:09:32.278 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 55896, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 4401, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 28896975 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [ 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: 1 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T11:09:32.279 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 10, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.280 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [] 2024-06-15T11:09:32.281 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:09:32.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:32.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-15T11:09:32.497 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:32.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-06-15T11:09:32.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-06-15T11:09:32.515 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-06-15T11:09:32.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:32.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-06-15T11:09:32.717 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:32.733 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:09:32.733 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 0, 2024-06-15T11:09:32.733 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 0, 2024-06-15T11:09:32.733 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 2, 2024-06-15T11:09:32.733 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 2, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 2, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 2, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 209715200, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 57120, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 656, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 8, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 56439, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 209658080, 2024-06-15T11:09:32.734 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 214748364800, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 214689873920, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 671744, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 111792, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 8799, 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 57793953 2024-06-15T11:09:32.735 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [], 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 20, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2024-06-15T11:09:32.736 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [], 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stdout: "network_ping_times": [] 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-06-15T11:09:32.737 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-06-15T11:09:32.943 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:09:32.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:09:32.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:09:32.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:09:32.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:09:32.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:09:33.270 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:09:33.271 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:09:33.272 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:09:33.273 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:09:33.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:09:33.275 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:09:33.276 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:09:33.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:09:33.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:09:33.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:09:33.278 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:09:33.279 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:09:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:09:33.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:09:33.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:09:33.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:33.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:33.299 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:09:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:09:33.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-06-15T11:09:33.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:09:33.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:09:33.302 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:09:33.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:09:33.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:09:33.302 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:09:33.304 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:09:33.305 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:09:33.306 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:09:33.306 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:09:33.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:09:33.307 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:09:33.308 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:09:33.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:09:33.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:09:33.309 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:09:33.310 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:09:33.315 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:09:33.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-06-15T11:09:33.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-06-15T11:09:33.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:33.317 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:33.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.62093 2024-06-15T11:09:33.319 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-06-15T11:09:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-06-15T11:09:33.320 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-06-15T11:09:33.321 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-06-15T11:09:33.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-06-15T11:09:33.322 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-06-15T11:09:33.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-06-15T11:09:33.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:09:33.694 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:09:33.695 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:09:33.695 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:33.695 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:33.696 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:33.696 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-06-15T11:09:33.922 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-06-15T11:09:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-06-15T11:09:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:09:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:09:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-06-15T11:09:33.924 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-06-15T11:09:33.925 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:09:33.925 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:09:33.925 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:09:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:09:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:33.926 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:09:33.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:09:33.927 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get fsid 2024-06-15T11:09:34.038 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-06-15T11:09:34.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-06-15T11:09:34.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-06-15T11:09:34.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-06-15T11:09:34.039 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-06-15T11:09:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-06-15T11:09:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-06-15T11:09:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-06-15T11:09:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-06-15T11:09:34.040 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:34.041 INFO:tasks.workunit.client.0.smithi116.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:34.041 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.62093/ceph-mon.a.asok 2024-06-15T11:09:34.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-06-15T11:09:34.041 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.62093/ceph-mon.a.asok config get mon_host 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-06-15T11:09:34.151 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-06-15T11:09:35.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-06-15T11:09:35.021 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:09:35.022 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:09:35.022 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:09:35.022 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:35.022 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:35.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:35.023 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:09:35.025 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-06-15T11:09:35.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:09:35.060 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-06-15T11:09:35.061 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:09:35.064 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:09:35.065 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:09:35.066 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:09:35.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e8e573f-f6dd-4111-b45a-63fd9a178479 2024-06-15T11:09:35.068 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2e8e573f-f6dd-4111-b45a-63fd9a178479' 2024-06-15T11:09:35.069 INFO:tasks.workunit.client.0.smithi116.stdout:add osd0 2e8e573f-f6dd-4111-b45a-63fd9a178479 2024-06-15T11:09:35.069 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:09:35.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvdm1mmxFfBRAA6ds2VgkGxXktYiTulgsNgA== 2024-06-15T11:09:35.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvdm1mmxFfBRAA6ds2VgkGxXktYiTulgsNgA=="}' 2024-06-15T11:09:35.088 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e8e573f-f6dd-4111-b45a-63fd9a178479 -i td/osd-scrub-repair/0/new.json 2024-06-15T11:09:35.324 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:09:35.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-06-15T11:09:35.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBvdm1mmxFfBRAA6ds2VgkGxXktYiTulgsNgA== --osd-uuid 2e8e573f-f6dd-4111-b45a-63fd9a178479 2024-06-15T11:09:35.376 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:35.378 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:35.379 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:35.379 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:09:35.379 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:09:35.380 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.378+0000 7f97766eb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-06-15T11:09:35.467 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:35.466+0000 7f97766eb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-06-15T11:09:41.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-06-15T11:09:41.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:09:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-06-15T11:09:41.172 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd0 key to auth repository 2024-06-15T11:09:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:09:41.476 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:09:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-06-15T11:09:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:09:41.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:09:41.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:09:41.478 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:09:41.514 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:41.514+0000 7f390340d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:41.541 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:41.542+0000 7f390340d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:41.542 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:41.542+0000 7f390340d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:09:41.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-06-15T11:09:41.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:09:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:09:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:09:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:09:41.723 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:09:41.723 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:41.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:09:41.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:41.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:41.963 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:42.098 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:42.098+0000 7f390340d7c0 -1 Falling back to public interface 2024-06-15T11:09:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:42.967 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:09:42.967 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:09:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:42.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:43.204 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:44.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:44.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:44.206 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:09:44.207 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:09:44.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:44.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:44.449 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:44.664 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:09:44.666+0000 7f390340d7c0 -1 osd.0 0 log_to_monitors true 2024-06-15T11:09:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:45.451 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:45.451 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:09:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:09:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:45.452 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:45.712 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:46.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:46.714 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:09:46.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:09:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:46.715 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:46.955 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:47.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:47.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:47.957 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:09:47.957 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:09:47.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:47.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:48.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:49.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:49.229 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:49.229 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:09:49.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:09:49.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:49.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:49.520 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:50.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:50.522 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:50.522 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:09:50.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:09:50.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:50.523 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:50.765 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:51.766 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:51.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:51.767 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:09:51.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:09:51.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:51.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:52.005 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:53.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:53.006 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:53.007 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:09:53.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:09:53.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:53.007 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:53.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:54.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:54.248 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:54.249 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:09:54.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:09:54.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:54.249 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:54.558 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:55.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:55.559 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:55.560 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:09:55.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:09:55.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:55.560 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:55.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:56.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:56.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:56.805 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:09:56.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:09:56.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:56.806 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:57.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:58.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:58.048 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:58.048 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:09:58.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:09:58.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:58.049 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:58.424 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:09:59.425 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:09:59.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:09:59.426 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:09:59.426 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:09:59.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:09:59.427 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:09:59.800 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:00.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:00.802 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:00.803 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:10:00.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:10:00.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:00.803 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:10:01.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:02.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:02.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:02.402 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:10:02.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:10:02.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:02.403 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:10:03.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:04.103 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:04.103 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:10:04.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:10:04.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:04.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:10:04.580 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:05.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:05.582 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:05.583 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:10:05.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:10:05.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:05.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:10:06.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:07.268 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:07.268 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:10:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:10:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:07.269 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:10:07.511 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/488027540,v1:127.0.0.1:6803/488027540] [v2:127.0.0.1:6804/488027540,v1:127.0.0.1:6805/488027540] exists,up 2e8e573f-f6dd-4111-b45a-63fd9a178479 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:10:07.513 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:10:07.514 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:10:07.515 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:10:07.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:10:07.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:10:07.516 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:10:07.517 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:10:07.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 2024-06-15T11:10:07.518 INFO:tasks.workunit.client.0.smithi116.stdout:add osd1 cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 2024-06-15T11:10:07.518 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cb4a63fb-3aa1-453c-b84d-c6c3fa62c533' 2024-06-15T11:10:07.519 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:10:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPdm1mqg8OIBAAThhvqHPtPWzVnY2/+tSlzg== 2024-06-15T11:10:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPdm1mqg8OIBAAThhvqHPtPWzVnY2/+tSlzg=="}' 2024-06-15T11:10:07.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 -i td/osd-scrub-repair/1/new.json 2024-06-15T11:10:07.816 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:10:07.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-06-15T11:10:07.833 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCPdm1mqg8OIBAAThhvqHPtPWzVnY2/+tSlzg== --osd-uuid cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 2024-06-15T11:10:07.870 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.870+0000 7f434176c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:07.901 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.902+0000 7f434176c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:07.902 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.902+0000 7f434176c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.902+0000 7f434176c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:10:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.902+0000 7f434176c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:10:07.903 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.902+0000 7f434176c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-06-15T11:10:07.971 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:07.970+0000 7f434176c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-06-15T11:10:13.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-06-15T11:10:13.761 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:10:13.763 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd1 key to auth repository 2024-06-15T11:10:13.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-06-15T11:10:13.763 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:10:14.029 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:10:14.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-06-15T11:10:14.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:10:14.030 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:10:14.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:10:14.031 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:10:14.064 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:14.066+0000 7f63f08897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:14.090 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:14.090+0000 7f63f08897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:14.090 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:14.090+0000 7f63f08897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:14.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-06-15T11:10:14.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:10:14.318 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:14.319 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:14.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:14.638 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:14.638+0000 7f63f08897c0 -1 Falling back to public interface 2024-06-15T11:10:15.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:15.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:15.555 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:10:15.555 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:10:15.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:15.556 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:15.796 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:16.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:16.798 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:16.799 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:10:16.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:10:16.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:16.799 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:17.036 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:17.193 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:17.194+0000 7f63f08897c0 -1 osd.1 0 log_to_monitors true 2024-06-15T11:10:18.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:18.038 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:18.038 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:10:18.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:10:18.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:18.039 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:18.363 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:19.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:19.365 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:19.365 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:10:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:10:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:19.366 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:19.607 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:20.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:20.609 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:20.609 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:10:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:10:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:20.610 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:20.856 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:21.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:21.858 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:21.859 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:10:21.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:10:21.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:21.859 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:22.197 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:23.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:23.199 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:23.199 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:10:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:10:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:23.200 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:23.458 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:24.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:24.460 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:24.461 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:10:24.461 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:10:24.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:24.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:24.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:25.702 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:25.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:25.703 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:10:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:10:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:25.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:26.015 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:27.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:27.017 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:27.018 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:10:27.018 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:10:27.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:27.019 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:27.257 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:28.258 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:28.259 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:10:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:10:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:28.259 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:28.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:29.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:29.549 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:29.549 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:10:29.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:10:29.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:29.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:29.973 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:30.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:30.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:30.975 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:10:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:10:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:30.976 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:31.399 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:32.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:32.401 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:32.402 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:10:32.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:10:32.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:32.402 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:33.143 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:34.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:34.145 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:34.146 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:10:34.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:10:34.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:34.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:34.828 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:35.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:35.830 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:35.830 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:10:35.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:10:35.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:35.831 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:36.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:37.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:37.724 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:37.724 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:10:37.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:10:37.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:37.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:38.106 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:39.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:39.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:39.109 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:10:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:10:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:39.109 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:39.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:40.971 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:40.971 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:10:40.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:10:40.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:40.972 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:10:41.224 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4043146324,v1:127.0.0.1:6811/4043146324] [v2:127.0.0.1:6812/4043146324,v1:127.0.0.1:6813/4043146324] exists,up cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 2024-06-15T11:10:41.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:10:41.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:10:41.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-06-15T11:10:41.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-06-15T11:10:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-06-15T11:10:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:10:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:10:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:10:41.234 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:10:41.235 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-06-15T11:10:41.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:10:41.236 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:10:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:10:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:10:41.236 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-06-15T11:10:41.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:10:41.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-06-15T11:10:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:10:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-06-15T11:10:41.239 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-06-15T11:10:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=877468ff-b4f8-48f7-a6fd-970685b9f202 2024-06-15T11:10:41.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 877468ff-b4f8-48f7-a6fd-970685b9f202' 2024-06-15T11:10:41.240 INFO:tasks.workunit.client.0.smithi116.stdout:add osd2 877468ff-b4f8-48f7-a6fd-970685b9f202 2024-06-15T11:10:41.240 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-06-15T11:10:41.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxdm1mFgM1DxAAIijcbKeDI2kLOe1ufcFU9A== 2024-06-15T11:10:41.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxdm1mFgM1DxAAIijcbKeDI2kLOe1ufcFU9A=="}' 2024-06-15T11:10:41.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 877468ff-b4f8-48f7-a6fd-970685b9f202 -i td/osd-scrub-repair/2/new.json 2024-06-15T11:10:41.602 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:10:41.619 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-06-15T11:10:41.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCxdm1mFgM1DxAAIijcbKeDI2kLOe1ufcFU9A== --osd-uuid 877468ff-b4f8-48f7-a6fd-970685b9f202 2024-06-15T11:10:41.657 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.657+0000 7f5787ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:41.686 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.685+0000 7f5787ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.689+0000 7f5787ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.689+0000 7f5787ab37c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:10:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.689+0000 7f5787ab37c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:10:41.688 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.689+0000 7f5787ab37c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-06-15T11:10:41.759 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:41.757+0000 7f5787ab37c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-06-15T11:10:47.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-06-15T11:10:47.817 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-06-15T11:10:47.819 INFO:tasks.workunit.client.0.smithi116.stdout:adding osd2 key to auth repository 2024-06-15T11:10:47.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-06-15T11:10:47.820 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-06-15T11:10:48.093 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.2 2024-06-15T11:10:48.093 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-06-15T11:10:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:10:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-06-15T11:10:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-06-15T11:10:48.094 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-06-15T11:10:48.127 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:48.125+0000 7fe2af5b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:48.158 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:48.157+0000 7fe2af5b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:48.160 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:48.161+0000 7fe2af5b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:10:48.350 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:10:48.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:48.351 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:48.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:48.702 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:48.701+0000 7fe2af5b07c0 -1 Falling back to public interface 2024-06-15T11:10:49.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:49.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:49.602 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:10:49.602 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:10:49.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:49.603 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:49.839 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:50.841 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:50.841 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:10:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:10:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:50.842 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:51.083 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:51.238 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:10:51.237+0000 7fe2af5b07c0 -1 osd.2 0 log_to_monitors true 2024-06-15T11:10:52.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:52.085 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:52.085 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:10:52.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:10:52.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:52.086 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:52.414 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:53.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:53.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:53.416 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-06-15T11:10:53.416 INFO:tasks.workunit.client.0.smithi116.stdout:4 2024-06-15T11:10:53.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:53.417 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:53.659 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:54.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:54.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:54.661 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-06-15T11:10:54.661 INFO:tasks.workunit.client.0.smithi116.stdout:5 2024-06-15T11:10:54.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:54.662 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:54.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:55.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:55.909 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:55.909 INFO:tasks.workunit.client.0.smithi116.stdout:6 2024-06-15T11:10:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-06-15T11:10:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:55.910 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:56.216 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:57.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:57.219 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:57.219 INFO:tasks.workunit.client.0.smithi116.stdout:7 2024-06-15T11:10:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-06-15T11:10:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:57.220 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:57.471 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:58.472 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:58.473 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:58.473 INFO:tasks.workunit.client.0.smithi116.stdout:8 2024-06-15T11:10:58.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-06-15T11:10:58.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:58.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:10:58.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:10:59.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:10:59.721 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:10:59.721 INFO:tasks.workunit.client.0.smithi116.stdout:9 2024-06-15T11:10:59.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-06-15T11:10:59.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:10:59.722 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:00.057 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:01.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:01.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:01.059 INFO:tasks.workunit.client.0.smithi116.stdout:10 2024-06-15T11:11:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-06-15T11:11:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:01.060 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:01.337 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:02.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:02.340 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:02.340 INFO:tasks.workunit.client.0.smithi116.stdout:11 2024-06-15T11:11:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-06-15T11:11:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:02.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:02.581 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:03.583 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:03.584 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:03.584 INFO:tasks.workunit.client.0.smithi116.stdout:12 2024-06-15T11:11:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-06-15T11:11:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:03.585 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:04.114 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:05.115 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:05.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:05.116 INFO:tasks.workunit.client.0.smithi116.stdout:13 2024-06-15T11:11:05.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-06-15T11:11:05.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:05.116 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:05.546 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:06.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:06.547 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:06.548 INFO:tasks.workunit.client.0.smithi116.stdout:14 2024-06-15T11:11:06.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-06-15T11:11:06.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:06.548 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:07.122 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:08.124 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-06-15T11:11:08.124 INFO:tasks.workunit.client.0.smithi116.stdout:15 2024-06-15T11:11:08.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:08.125 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:08.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:09.903 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:09.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:09.904 INFO:tasks.workunit.client.0.smithi116.stdout:16 2024-06-15T11:11:09.904 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-06-15T11:11:09.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:09.905 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:10.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:11.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:11.395 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:11.396 INFO:tasks.workunit.client.0.smithi116.stdout:17 2024-06-15T11:11:11.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-06-15T11:11:11.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:11.396 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:11.872 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:12.873 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:12.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:12.874 INFO:tasks.workunit.client.0.smithi116.stdout:18 2024-06-15T11:11:12.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-06-15T11:11:12.874 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:12.875 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:13.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:14.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:14.435 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:14.436 INFO:tasks.workunit.client.0.smithi116.stdout:19 2024-06-15T11:11:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-06-15T11:11:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:14.436 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:14.701 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:11:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:11:15.703 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:11:15.704 INFO:tasks.workunit.client.0.smithi116.stdout:20 2024-06-15T11:11:15.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-06-15T11:11:15.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:11:15.704 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-06-15T11:11:15.944 INFO:tasks.workunit.client.0.smithi116.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2122793592,v1:127.0.0.1:6819/2122793592] [v2:127.0.0.1:6820/2122793592,v1:127.0.0.1:6821/2122793592] exists,up 877468ff-b4f8-48f7-a6fd-970685b9f202 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-06-15T11:11:15.945 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-06-15T11:11:16.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-06-15T11:11:16.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-06-15T11:11:17.050 INFO:tasks.workunit.client.0.smithi116.stderr:pool 'testpool' created 2024-06-15T11:11:17.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-06-15T11:11:18.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-06-15T11:11:18.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-06-15T11:11:18.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:11:18.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:11:18.075 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:11:18.076 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:11:18.077 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:11:18.077 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:11:18.077 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:11:18.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:11:18.078 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:11:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:11:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:11:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:11:18.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:11:18.214 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:11:18.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:11:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:11:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:11:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:11:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:11:18.551 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:11:18.552 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:11:18.697 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-06-15T11:11:18.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-06-15T11:11:18.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-06-15T11:11:18.698 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:11:18.698 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:11:18.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-06-15T11:11:18.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-06-15T11:11:18.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672971' 2024-06-15T11:11:18.840 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:11:18.841 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:11:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-06-15T11:11:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-06-15T11:11:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672971 2-64424509445' 2024-06-15T11:11:18.981 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:11:18.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-06-15T11:11:18.982 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:11:18.982 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:11:18.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-06-15T11:11:18.983 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:11:18.984 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-06-15T11:11:18.985 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 21474836503 2024-06-15T11:11:18.985 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-06-15T11:11:18.985 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:11:19.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-06-15T11:11:19.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:11:20.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:11:20.214 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:11:20.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-06-15T11:11:20.456 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:11:20.457 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-06-15T11:11:20.458 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:11:20.459 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:11:20.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-06-15T11:11:20.460 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:11:20.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-06-15T11:11:20.462 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 42949672971 2024-06-15T11:11:20.462 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-06-15T11:11:20.462 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:11:20.713 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-06-15T11:11:20.714 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:11:20.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-06-15T11:11:20.714 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:11:20.716 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:11:20.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-06-15T11:11:20.717 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:11:20.718 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-06-15T11:11:20.719 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509445 2024-06-15T11:11:20.719 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-06-15T11:11:20.719 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:11:20.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-06-15T11:11:20.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:11:20.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:11:20.959 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:11:21.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:11:21.227 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:11:21.228 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:11:21.499 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:11:21.500 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:11:21.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:11:21.501 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:11:21.755 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-06-15T11:11:21.756 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-06-15T11:11:21.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:21.757 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:21.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:21.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:22.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:22.154 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:22.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:22.454 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:22.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:22.680 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:22.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:22.907 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:23.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:23.091 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:23.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:23.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:23.535 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:23.536 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:23.769 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:23.770 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:23.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:23.988 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:24.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:24.182 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:24.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:24.434 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:24.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:24.625 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:24.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:24.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:25.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:25.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:25.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:25.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:25.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:25.498 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:25.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:25.682 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:25.925 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:26.169 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:26.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:26.429 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:26.670 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:26.671 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:26.888 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:26.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:27.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:27.099 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:27.276 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:27.277 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:27.526 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:27.777 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:27.778 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:27.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:27.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:28.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-06-15T11:11:28.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-06-15T11:11:28.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-06-15T11:11:28.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-06-15T11:11:28.467 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-06-15T11:11:28.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-06-15T11:11:28.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T11:11:28.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T11:11:28.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T11:11:28.468 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T11:11:28.715 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-06-15T11:11:28.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-06-15T11:11:28.716 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-06-15T11:11:28.953 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-06-15T11:11:28.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-06-15T11:11:28.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-06-15T11:11:28.954 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-06-15T11:11:28.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-06-15T11:11:28.955 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-06-15T11:11:29.188 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-06-15T11:11:29.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-06-15T11:11:29.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-06-15T11:11:29.189 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-06-15T11:11:29.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-06-15T11:11:29.190 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-06-15T11:11:29.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-06-15T11:11:29.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-06-15T11:11:29.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:11:29.475 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:11:29.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:11:29.475 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:11:29.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:11:29.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:11:29.786 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-06-15T11:11:29.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:11:29.787 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:11:29.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:11:29.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:11:29.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:11:31.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:11:31.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-06-15T11:11:31.104 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:31.106 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:31.108 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-06-15T11:11:32.353 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-06-15T11:11:32.889 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:32.890 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:32.891 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-06-15T11:11:34.017 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:104778fc:::obj2:head# 2024-06-15T11:11:34.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:34.554 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:34.557 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-06-15T11:11:35.393 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-06-15T11:11:35.930 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:35.931 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-06-15T11:11:35.932 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:35.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-06-15T11:11:35.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:35.933 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:35.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:35.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:35.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:35.934 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-06-15T11:11:36.763 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-06-15T11:11:37.298 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:37.298 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-06-15T11:11:37.300 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:37.301 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-06-15T11:11:38.014 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:c52c9666:::obj5:head# 2024-06-15T11:11:38.550 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:38.551 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:38.553 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-06-15T11:11:39.417 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:233a42c1:::obj6:head# 2024-06-15T11:11:39.962 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:39.963 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-06-15T11:11:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-06-15T11:11:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:39.964 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:39.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:39.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:39.965 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-06-15T11:11:40.636 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:9a09113a:::obj7:head# 2024-06-15T11:11:41.168 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:41.169 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:41.172 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-06-15T11:11:41.846 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:a557efb1:::obj8:head# 2024-06-15T11:11:42.390 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:42.391 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-06-15T11:11:42.392 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:42.393 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-06-15T11:11:43.073 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:fe351e27:::obj9:head# 2024-06-15T11:11:43.618 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:43.619 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-06-15T11:11:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-06-15T11:11:43.621 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:43.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:43.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:43.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:43.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:43.622 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-06-15T11:11:44.459 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:d1337354:::obj10:head# 2024-06-15T11:11:44.994 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:44.994 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-06-15T11:11:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:44.996 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:44.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-06-15T11:11:45.711 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-06-15T11:11:46.501 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:46.502 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-06-15T11:11:46.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:46.504 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:46.505 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-06-15T11:11:47.189 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-06-15T11:11:47.975 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:47.976 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-06-15T11:11:47.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:47.977 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:47.978 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-06-15T11:11:48.634 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-06-15T11:11:49.170 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:49.171 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-06-15T11:11:49.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:49.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-06-15T11:11:49.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:49.173 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:49.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:49.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:49.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:49.174 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-06-15T11:11:49.778 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:e0a44829:::obj14:head# 2024-06-15T11:11:50.314 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:50.315 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-06-15T11:11:50.316 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:50.317 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-06-15T11:11:50.959 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:ab946124:::obj15:head# 2024-06-15T11:11:51.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:51.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:51.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-06-15T11:11:52.154 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-06-15T11:11:52.690 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:52.691 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-06-15T11:11:52.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:52.693 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-06-15T11:11:52.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:52.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:52.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:52.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:52.694 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:52.695 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-06-15T11:11:53.424 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-06-15T11:11:53.958 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:53.958 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-06-15T11:11:53.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:53.960 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:53.961 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-06-15T11:11:54.669 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:b9836a99:::obj18:head# 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:55.207 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:55.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:55.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:55.208 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-06-15T11:11:55.803 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-06-15T11:11:56.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:56.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:56.341 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:56.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:56.342 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-06-15T11:11:56.997 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:24b5611b:::obj20:head# 2024-06-15T11:11:57.534 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:57.535 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-06-15T11:11:57.537 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:11:57.538 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-06-15T11:11:58.357 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-06-15T11:11:58.898 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:11:58.899 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:11:58.901 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:11:58.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:11:58.902 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-06-15T11:11:59.702 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:c0c49def:::obj22:head# 2024-06-15T11:12:00.250 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:12:00.251 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-06-15T11:12:00.253 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:12:00.254 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-06-15T11:12:00.937 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:242c40e0:::obj23:head# 2024-06-15T11:12:01.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:12:01.478 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:12:01.480 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-06-15T11:12:02.212 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:7eacede5:::obj24:head# 2024-06-15T11:12:02.767 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:12:02.767 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:12:02.768 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-06-15T11:12:03.507 INFO:tasks.workunit.client.0.smithi116.stdout:remove 0#1:7109478f:::obj25:head# 2024-06-15T11:12:04.042 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-06-15T11:12:04.043 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-06-15T11:12:04.045 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:12:04.046 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-06-15T11:12:04.701 INFO:tasks.workunit.client.0.smithi116.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-06-15T11:12:05.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:12:05.238 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:12:05.239 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:12:05.240 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:12:05.241 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:12:05.242 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-06-15T11:12:05.243 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.1 2024-06-15T11:12:05.243 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-06-15T11:12:05.244 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:12:05.244 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-06-15T11:12:05.245 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-06-15T11:12:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:12:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:12:05.246 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:12:05.285 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:05.284+0000 7faccca087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:05.317 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:05.316+0000 7faccca087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:05.318 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:05.316+0000 7faccca087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:12:05.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:05.486 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:12:05.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:12:05.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:05.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:12:05.725 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:05.862 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:05.860+0000 7faccca087c0 -1 Falling back to public interface 2024-06-15T11:12:06.726 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:06.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:06.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:12:06.727 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:12:06.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:06.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:12:06.966 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:07.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:07.968 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:07.968 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:12:07.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:12:07.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:07.969 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:12:08.211 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:08.413 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:08.412+0000 7faccca087c0 -1 osd.1 23 log_to_monitors true 2024-06-15T11:12:09.212 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:09.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:09.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:12:09.213 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:12:09.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:09.213 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-06-15T11:12:09.488 INFO:tasks.workunit.client.0.smithi116.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/181584662,v1:127.0.0.1:6803/181584662] [v2:127.0.0.1:6804/181584662,v1:127.0.0.1:6805/181584662] exists,up cb4a63fb-3aa1-453c-b84d-c6c3fa62c533 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-06-15T11:12:09.490 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-06-15T11:12:09.491 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-06-15T11:12:09.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-06-15T11:12:09.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-06-15T11:12:09.492 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-06-15T11:12:09.493 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-06-15T11:12:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-06-15T11:12:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-06-15T11:12:09.494 INFO:tasks.workunit.client.0.smithi116.stdout:start osd.0 2024-06-15T11:12:09.494 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-06-15T11:12:09.495 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f2d71c0f-4e36-4969-9763-a007f8142d8c --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.62093/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-06-15T11:12:09.495 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-06-15T11:12:09.496 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-06-15T11:12:09.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-06-15T11:12:09.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-06-15T11:12:09.497 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-06-15T11:12:09.533 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:09.532+0000 7fcc4684c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:09.535 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:09.532+0000 7fcc4684c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:09.536 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:09.536+0000 7fcc4684c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:09.728 INFO:tasks.workunit.client.0.smithi116.stdout:0 2024-06-15T11:12:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-06-15T11:12:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:09.729 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:12:09.987 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:10.073 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:10.072+0000 7fcc4684c7c0 -1 Falling back to public interface 2024-06-15T11:12:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:10.989 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:10.989 INFO:tasks.workunit.client.0.smithi116.stdout:1 2024-06-15T11:12:10.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-06-15T11:12:10.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:10.990 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:12:11.230 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:12.232 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:12.233 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:12.235 INFO:tasks.workunit.client.0.smithi116.stdout:2 2024-06-15T11:12:12.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-06-15T11:12:12.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:12.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:12:12.474 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-06-15T11:12:12.626 INFO:tasks.workunit.client.0.smithi116.stderr:2024-06-15T11:12:12.624+0000 7fcc4684c7c0 -1 osd.0 21 log_to_monitors true 2024-06-15T11:12:13.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-06-15T11:12:13.476 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-06-15T11:12:13.476 INFO:tasks.workunit.client.0.smithi116.stdout:3 2024-06-15T11:12:13.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-06-15T11:12:13.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-06-15T11:12:13.477 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-06-15T11:12:13.757 INFO:tasks.workunit.client.0.smithi116.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/1467989250,v1:127.0.0.1:6811/1467989250] [v2:127.0.0.1:6812/1467989250,v1:127.0.0.1:6813/1467989250] exists,up 2e8e573f-f6dd-4111-b45a-63fd9a178479 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:12:13.758 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:12:13.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:12:13.759 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:12:13.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:12:13.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:12:13.760 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:12:13.877 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:12:14.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:12:14.133 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:12:14.133 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:12:14.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:12:14.133 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:14.134 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:12:14.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-06-15T11:12:14.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-06-15T11:12:14.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-06-15T11:12:14.280 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:14.281 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:12:14.432 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116995 2024-06-15T11:12:14.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116995 2024-06-15T11:12:14.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-06-15T11:12:14.433 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:14.433 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:12:14.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-06-15T11:12:14.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-06-15T11:12:14.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509457' 2024-06-15T11:12:14.570 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:14.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-06-15T11:12:14.571 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:14.572 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:12:14.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-06-15T11:12:14.574 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:14.575 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-06-15T11:12:14.575 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 128849018882 2024-06-15T11:12:14.576 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-06-15T11:12:14.576 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:14.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-06-15T11:12:14.805 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:12:15.807 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:12:15.808 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:16.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-06-15T11:12:16.052 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:16.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116995 2024-06-15T11:12:16.053 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:16.055 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:12:16.056 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116995 2024-06-15T11:12:16.057 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:16.058 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116995 2024-06-15T11:12:16.058 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 115964116995 2024-06-15T11:12:16.059 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-06-15T11:12:16.059 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:12:16.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116995 -lt 115964116995 2024-06-15T11:12:16.308 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:16.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:16.309 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-06-15T11:12:16.310 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:12:16.311 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-06-15T11:12:16.312 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:16.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-06-15T11:12:16.313 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509457 2024-06-15T11:12:16.313 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-06-15T11:12:16.314 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:12:16.543 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-06-15T11:12:16.544 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:12:16.545 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:12:16.546 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:12:16.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:12:16.797 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:12:16.798 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:12:16.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:12:16.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:12:16.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:12:16.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:12:16.799 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:12:17.051 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:12:17.051 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:12:17.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:12:17.052 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:12:17.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:12:17.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:12:17.338 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:12:17.339 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:12:17.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-06-15T11:11:16.935087+0000 2024-06-15T11:12:17.592 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-06-15T11:12:17.796 INFO:tasks.workunit.client.0.smithi116.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-06-15T11:12:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-06-15T11:11:16.935087+0000 2024-06-15T11:12:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-06-15T11:12:17.811 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-06-15T11:11:16.935087+0000 2024-06-15T11:12:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-06-15T11:12:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-06-15T11:12:17.812 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:12:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T11:12:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:12:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:12:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:12:17.813 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:12:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:11:16.935087+0000 '>' 2024-06-15T11:11:16.935087+0000 2024-06-15T11:12:18.021 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-06-15T11:12:19.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-06-15T11:12:19.023 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-06-15T11:12:19.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-06-15T11:12:19.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-06-15T11:12:19.024 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-06-15T11:12:19.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-06-15T11:12:19.025 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-06-15T11:12:18.414311+0000 '>' 2024-06-15T11:11:16.935087+0000 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-06-15T11:12:19.266 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-06-15T11:12:19.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-06-15T11:12:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-06-15T11:12:19.267 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-06-15T11:12:19.267 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-06-15T11:12:19.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-06-15T11:12:19.268 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-06-15T11:12:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-06-15T11:12:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-06-15T11:12:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-06-15T11:12:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-06-15T11:12:19.387 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:12:19.388 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:12:19.641 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:12:19.641 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:12:19.641 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:12:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:12:19.642 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:19.642 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:12:19.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018884 2024-06-15T11:12:19.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018884 2024-06-15T11:12:19.815 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018884' 2024-06-15T11:12:19.816 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:19.816 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:12:19.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116997 2024-06-15T11:12:19.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116997 2024-06-15T11:12:19.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018884 1-115964116997' 2024-06-15T11:12:19.956 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:19.957 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:12:20.100 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-06-15T11:12:20.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-06-15T11:12:20.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018884 1-115964116997 2-64424509459' 2024-06-15T11:12:20.101 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:20.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018884 2024-06-15T11:12:20.101 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:20.102 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:12:20.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018884 2024-06-15T11:12:20.103 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:20.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018884 2024-06-15T11:12:20.105 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 128849018884 2024-06-15T11:12:20.105 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2024-06-15T11:12:20.105 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018883 -lt 128849018884 2024-06-15T11:12:20.353 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:12:21.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:12:21.355 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:21.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018884 2024-06-15T11:12:21.595 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:21.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116997 2024-06-15T11:12:21.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:21.597 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:12:21.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116997 2024-06-15T11:12:21.598 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:21.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116997 2024-06-15T11:12:21.600 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116997' 2024-06-15T11:12:21.600 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 115964116997 2024-06-15T11:12:21.600 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:12:21.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116997 -lt 115964116997 2024-06-15T11:12:21.829 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:21.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-06-15T11:12:21.830 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:21.832 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:12:21.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-06-15T11:12:21.834 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:21.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-06-15T11:12:21.837 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-06-15T11:12:21.838 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509459 2024-06-15T11:12:21.838 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:12:22.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-06-15T11:12:22.082 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-06-15T11:12:22.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:12:22.083 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:12:22.355 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-06-15T11:12:22.356 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-06-15T11:12:22.356 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-06-15T11:12:22.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-06-15T11:12:22.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-06-15T11:12:22.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-06-15T11:12:22.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-06-15T11:12:22.357 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-06-15T11:12:22.594 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-06-15T11:12:22.595 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-06-15T11:12:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-06-15T11:12:22.596 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-06-15T11:12:22.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-06-15T11:12:22.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-06-15T11:12:22.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-06-15T11:12:22.852 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-06-15T11:12:23.075 INFO:tasks.workunit.client.0.smithi116.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-06-15T11:12:23.076 INFO:tasks.workunit.client.0.smithi116.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-06-15T11:12:19.599080+0000 21'30 31:134 [1,0,2] 1 [1,0,2] 1 21'30 2024-06-15T11:12:18.414311+0000 21'30 2024-06-15T11:12:18.414311+0000 0 1 periodic scrub scheduled @ 2024-06-16T11:12:18.414311+0000 17 0 2024-06-15T11:12:23.076 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-06-15T11:12:23.076 INFO:tasks.workunit.client.0.smithi116.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-06-15T11:12:23.076 INFO:tasks.workunit.client.0.smithi116.stderr:dumped pgs 2024-06-15T11:12:23.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-06-15T11:12:23.089 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-06-15T11:12:23.090 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-06-15T11:12:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-06-15T11:12:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:1 2024-06-15T11:12:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:2' 2024-06-15T11:12:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-06-15T11:12:23.323 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:23.324 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-06-15T11:12:23.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018886 2024-06-15T11:12:23.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018886 2024-06-15T11:12:23.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886' 2024-06-15T11:12:23.512 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:23.512 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-06-15T11:12:23.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116998 2024-06-15T11:12:23.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116998 2024-06-15T11:12:23.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886 1-115964116998' 2024-06-15T11:12:23.646 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-06-15T11:12:23.647 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-06-15T11:12:23.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-06-15T11:12:23.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-06-15T11:12:23.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018886 1-115964116998 2-64424509461' 2024-06-15T11:12:23.785 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:23.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018886 2024-06-15T11:12:23.786 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:23.787 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-06-15T11:12:23.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018886 2024-06-15T11:12:23.788 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:23.789 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018886 2024-06-15T11:12:23.790 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.0 seq 128849018886 2024-06-15T11:12:23.790 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018886' 2024-06-15T11:12:23.790 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:24.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018886 2024-06-15T11:12:24.008 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:12:25.009 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-06-15T11:12:25.010 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:25.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018886 2024-06-15T11:12:25.236 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-06-15T11:12:26.237 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-06-15T11:12:26.238 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-06-15T11:12:26.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018886 2024-06-15T11:12:26.479 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:26.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116998 2024-06-15T11:12:26.480 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:26.483 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-06-15T11:12:26.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116998 2024-06-15T11:12:26.484 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:26.485 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116998 2024-06-15T11:12:26.485 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.1 seq 115964116998 2024-06-15T11:12:26.486 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-06-15T11:12:26.486 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-06-15T11:12:26.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116999 -lt 115964116998 2024-06-15T11:12:26.727 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-06-15T11:12:26.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-06-15T11:12:26.728 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-06-15T11:12:26.730 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-06-15T11:12:26.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-06-15T11:12:26.731 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-06-15T11:12:26.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-06-15T11:12:26.733 INFO:tasks.workunit.client.0.smithi116.stdout:waiting osd.2 seq 64424509461 2024-06-15T11:12:26.733 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-06-15T11:12:26.733 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-06-15T11:12:26.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-06-15T11:12:26.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-15T11:12:26.997 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-06-15T11:12:27.147 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes": 210, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects": 30, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_clones": 0, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_object_copies": 90, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing_on_primary": 0, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_missing": 0, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_degraded": 0, 2024-06-15T11:12:27.148 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_misplaced": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_unfound": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_dirty": 30, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_whiteouts": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_read_kb": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write": 30, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_write_kb": 30, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_scrub_errors": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shallow_scrub_errors": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_deep_scrub_errors": 0, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_recovered": 26, 2024-06-15T11:12:27.149 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_recovered": 182, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_keys_recovered": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_omap": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_hit_set_archive": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_bytes_hit_set_archive": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_kb": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_kb": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_promote": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_high": 0, 2024-06-15T11:12:27.150 INFO:tasks.workunit.client.0.smithi116.stdout: "num_flush_mode_low": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_some": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_evict_mode_full": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_pinned": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_legacy_snapsets": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_large_omap_objects": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_manifest": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_bytes": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_omap_keys": 0, 2024-06-15T11:12:27.151 INFO:tasks.workunit.client.0.smithi116.stdout: "num_objects_repaired": 26 2024-06-15T11:12:27.153 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:12:27.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-06-15T11:12:27.154 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-06-15T11:12:27.292 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-06-15T11:12:27.293 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-06-15T11:12:27.293 INFO:tasks.workunit.client.0.smithi116.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-06-15T11:12:27.294 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-06-15T11:12:27.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-06-15T11:12:27.295 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-06-15T11:12:27.295 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-06-15T11:12:27.296 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-06-15T11:12:27.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-15T11:12:27.297 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-06-15T11:12:27.504 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:12:27.520 INFO:tasks.workunit.client.0.smithi116.stdout:{ 2024-06-15T11:12:27.520 INFO:tasks.workunit.client.0.smithi116.stdout: "osd": 0, 2024-06-15T11:12:27.520 INFO:tasks.workunit.client.0.smithi116.stdout: "up_from": 30, 2024-06-15T11:12:27.520 INFO:tasks.workunit.client.0.smithi116.stdout: "seq": 128849018886, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "num_pgs": 1, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "num_osds": 1, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_osds": 1, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "num_per_pool_omap_osds": 1, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "kb": 104857600, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used": 28480, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_data": 376, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_omap": 4, 2024-06-15T11:12:27.521 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_used_meta": 28091, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "kb_avail": 104829120, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "statfs": { 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "total": 107374182400, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "available": 107345018880, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "internally_reserved": 0, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "allocated": 385024, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "data_stored": 206193, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed": 0, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_allocated": 0, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "data_compressed_original": 0, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "omap_allocated": 4402, 2024-06-15T11:12:27.522 INFO:tasks.workunit.client.0.smithi116.stdout: "internal_metadata": 28765902 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: "hb_peers": [ 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: 1, 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: 2 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: ], 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: "snap_trim_queue_len": 0, 2024-06-15T11:12:27.523 INFO:tasks.workunit.client.0.smithi116.stdout: "num_snap_trimming": 0, 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "num_shards_repaired": 26, 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "op_queue_age_hist": { 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "histogram": [], 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "upper_bound": 1 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "perf_stat": { 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ms": 0, 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ms": 0, 2024-06-15T11:12:27.524 INFO:tasks.workunit.client.0.smithi116.stdout: "commit_latency_ns": 0, 2024-06-15T11:12:27.525 INFO:tasks.workunit.client.0.smithi116.stdout: "apply_latency_ns": 0 2024-06-15T11:12:27.525 INFO:tasks.workunit.client.0.smithi116.stdout: }, 2024-06-15T11:12:27.525 INFO:tasks.workunit.client.0.smithi116.stdout: "alerts": [] 2024-06-15T11:12:27.525 INFO:tasks.workunit.client.0.smithi116.stdout:} 2024-06-15T11:12:27.525 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-06-15T11:12:27.526 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-06-15T11:12:27.736 INFO:tasks.workunit.client.0.smithi116.stderr:dumped all 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-06-15T11:12:27.753 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-06-15T11:12:27.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-06-15T11:12:27.754 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-06-15T11:12:27.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-06-15T11:12:27.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-06-15T11:12:27.754 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-06-15T11:12:28.067 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-06-15T11:12:28.068 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-06-15T11:12:28.069 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-06-15T11:12:28.070 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-06-15T11:12:28.071 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-06-15T11:12:28.072 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-06-15T11:12:28.072 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-06-15T11:12:28.073 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:12:28.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-06-15T11:12:28.074 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-06-15T11:12:28.075 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T11:12:28.076 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:12:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-06-15T11:12:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-06-15T11:12:28.078 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-06-15T11:12:28.080 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-06-15T11:12:28.081 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-06-15T11:12:28.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-06-15T11:12:28.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-06-15T11:12:28.096 INFO:tasks.workunit.client.0.smithi116.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.62093 2024-06-15T11:12:28.096 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.62093 2024-06-15T11:12:28.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-06-15T11:12:28.097 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-06-15T11:12:28.098 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-06-15T11:12:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T11:12:28.099 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-06-15T11:12:28.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-15T11:12:28.419 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-15T11:12:28.613 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1b99e45e600e49c381f93f740a179bf4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_db0330b1e4e2470d52b750e251e55a522b4f7d69/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-06-15T11:12:28.616 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-15T11:12:28.627 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-15T11:12:28.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T11:12:28.668 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-15T11:12:28.679 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-15T11:12:28.679 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T11:12:28.700 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-06-15T11:12:28.700 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-06-15T11:12:28.700 INFO:teuthology.orchestra.run.smithi116.stdout:+hv01.front.sepi 67.205.162.81 3 u 93 128 377 0.100 -2.799 0.203 2024-06-15T11:12:28.700 INFO:teuthology.orchestra.run.smithi116.stdout:+hv02.front.sepi 63.231.80.2 3 u 83 128 377 0.119 +1.608 0.155 2024-06-15T11:12:28.700 INFO:teuthology.orchestra.run.smithi116.stdout:*hv03.front.sepi 74.6.168.72 3 u 43 128 377 0.120 +0.071 0.116 2024-06-15T11:12:28.701 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-06-15T11:12:28.701 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-15T11:12:28.714 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-15T11:12:28.714 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-15T11:12:28.728 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-15T11:12:28.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-15T11:12:28.756 INFO:teuthology.task.internal:Duration was 6297.809137 seconds 2024-06-15T11:12:28.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-15T11:12:28.766 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-15T11:12:28.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T11:12:28.808 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-15T11:12:28.808 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-06-15T11:12:28.808 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T11:12:28.861 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-15T11:12:28.861 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T11:12:28.947 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-15T11:12:28.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T11:12:29.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-15T11:12:29.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-15T11:12:29.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T11:12:29.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-15T11:12:29.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:12:29.117 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-06-15T11:12:29.139 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T11:12:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T11:12:29.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-15T11:12:29.193 INFO:teuthology.task.internal:Transferring archived files... 2024-06-15T11:12:29.195 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:48:07-rados-squid-release-distro-default-smithi/7756600/remote/smithi116 2024-06-15T11:12:29.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T11:12:30.132 INFO:teuthology.task.internal:Removing archive directory... 2024-06-15T11:12:30.132 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T11:12:30.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-15T11:12:30.188 INFO:teuthology.task.internal:Not uploading archives. 2024-06-15T11:12:30.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-15T11:12:30.198 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-15T11:12:30.198 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T11:12:30.228 INFO:teuthology.orchestra.run.smithi116.stdout: 658642 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 15 11:12 /home/ubuntu/cephtest 2024-06-15T11:12:30.228 INFO:teuthology.orchestra.run.smithi116.stdout: 671751 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 15 09:36 /home/ubuntu/cephtest/mnt.0 2024-06-15T11:12:30.228 INFO:teuthology.orchestra.run.smithi116.stdout: 673778 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 15 09:47 /home/ubuntu/cephtest/mnt.0/client.0 2024-06-15T11:12:30.228 INFO:teuthology.orchestra.run.smithi116.stdout: 788936 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jun 15 09:47 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-15T11:12:30.229 INFO:teuthology.orchestra.run.smithi116.stdout: 788937 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 15 11:12 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-06-15T11:12:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T11:12:30.230 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-06-15T11:12:30.230 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-15T11:12:30.235 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-15T11:12:30.246 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-15T11:12:30.257 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-06-15T11:12:30.262 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 6297.80913734436 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi116 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=db0330b1e4e2470d52b750e251e55a522b4f7d69 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1b99e45e600e49c381f93f740a179bf4 status: fail success: false 2024-06-15T11:12:30.262 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T11:12:30.354 INFO:teuthology.run:FAIL